NAS Server Testing, Part 2: IOZone

Using IOzone to create a synthetic  workload and possibly hurt your server

 

You can get just about as sophisticated as you like on testing an NAS server,  but like most things computer, the 80/20 rule applies. In this case we can get  to 80% of what we need to know about the total NAS server performance profile  while only using about 20% of the features some of these test tools have. That  is good, because it seems you could spend your professional career doing  performance and capacity planning and profiling. The folks that attend CMG (Computer Measurement Group)  conferences do spend most of their time  doing this. But we are an R&D Support group, and this is only a small part  of what we do. We try to be good at it, but not perfect. Good enough that the  servers perform well and are stable. But we do not have them modeled to 3  decimal places. There are new versions of Linux to install, and unannounced  hardware to open up and peek at!

We make the assumption going in to the testing that read performance will be  either the same or better than write performance. We therefore only ever really  care about write performance, and so only use the "write" side of the tool set  for some of the tests. Reads are generally cached all over the place, in the  server, the client, the hardware, and the various OS's... sometimes even the  network. In practice I have never seen reads be lower than writes, as the internal doc below  notes.

 

IOzone is a terrific  synthetic workload generator. We only use a tiny part of what it can do. I'll  break out some of the the whys after wherefores after the commands listed  below

 

From our internal WIKI doc, these are the commands we use:


 

#1 Basic function test

Use a couple of "well behaved NFS clients" (Solaris and Linux) to run a  simple iozone test against the NFS server. Also, verify locking is working via  our "locktest" tool.

  1. iozone -s 100m -r 32k -i 0 -w -f /nfs_mount_point/basic.ioz

  2. locktest /nfs_mount_point/basic.ioz - BMC simple lock test

  3. iozone -s 100m -r 32k -i 1 -f  /nfs_mount_point/basic.ioz

 

#2 Full function test

<subject of future column: move along. Nothing  to see here >

 

#3 Client platform test

Same as the basic test, except repeated on all clients and  use the automounter path to access the remote NFS filesystem.

  • iozone -s 300m -r 32k -i 0 -w -f /nfs_mount_point/full.ioz

    • iozone -B -s 100m -r 32k -i 0 -w -f ?:\basic.ioz <MS Windows  syntax>

  • locktest /nfs_mount_point/full.ioz - BMC simple locktest

  • iozone -s 300m -r 32k -i 1 -f /nfs_mount_point/full.ioz

 

#4 Load test

Using a selected set of NFS clients (probably around 20 fast clients,  hopefully gigabit attached) run multiple, parallel, simultanous iozone streams  to the NFS server to generate a large NFS write workload. Network throughput  at the NFS server will be measured to record NFS write performance.

  • iozone -s 100m -r 32k -i 0 -f /nfs_mount_point/load.ioz

 

NOTE: NFS write performance is evaluated and measured, as  NFS read performance is always many factors better than NFS write performance.

 

 

#5 Burn test

Using a selected set of NFS clients, run a full pass of iozone against the  NFS server.

    • iozone -a -f /nfs_mount_point/burn.ioz


 

Locktest

Our locktest is a very simple little C program that we can compile on any  platform. It calls the C lockf function, and then waits for you to hit enter and  then unlocks the file. All we are doing it making sure that the client can talk  to the server lockd. It can also be used on 2 different client to be sure locks  are honored: lock a file, go else where and try to get a lock on the locked  file.


A note about IOzone on MS Windows

We also run IOzone on MS Windows under Cygwin, but use the -B option to speed it up, otherwise  performance under Cygwin is rather slow, and we are measuring Cygwin, not the  file server. This is the bit from the man page about -B

 

-B
Use mmap() files. This causes all  of the temporary files being measured to be created and accessed with the mmap()  interface.
Some applications prefer to treat files as arrays of  memory.
These applications mmap() the file and then just access the  ar-
ray with loads and stores to perform file I/O.

 

I mention it here because... well.. that man entry would not normally clue me  in that is what you need to do to speed up things.

 

Taking it in sequence

 

iozone -s 100m -r 32k -i 0 -w -f  /nfs_mount_point/basic.ioz

 

This one creates a file of 100 Meg, using 32k blocks. -i 0 being the write  test, -w meaning "leave the file there", and the -F being the path to the NFS  mounted file system


locktest /nfs_mount_point/basic.ioz -  BMC simple lock test

 

Now we lockf the file we just created, and wait for enter to be pressed, then  unlock it.

 

iozone -s 100m -r 32k -i 1 -f  /nfs_mount_point/basic.ioz

 

This one runs test 1 via the "-i 1" options, which is just the read test.  This one is mostly just to see if our assumption about reads being faster than  writes is really true.

 

iozone -a -f /nfs_mount_point/burn.ioz

 

The -a option runs a whole automatic  sequence of tests, stepping up in file and buffer sizes, and using various read  and write strategies. Since we started it from a bunch of clients, and they are  all different speeds, things are happening at different times for each client  and the server has to deal with all of it at  the same time.

 

Measuring the server side

 

If the server we are testing supports BMC Performance Manager, we can use  that to watch how the server beatdown is going from the servers point of view.  Once the network pipe is full, we won't really be able to push the server any  harder, although if there are multiple network interfaces we can be really cruel  and drive I/O to all of them at the same time to see what happens.

 

From our Netperf testing we have a pretty good idea what the network can do here. Now we can see  how adding in the protocols and disk subsystems affect things. In general we'll  see a maximum speed for 1 stream (Our Apple Xserves are the fastest single  client systems we have, NFS or CIFS), what the server max is (generally a few  fast clients give us this number) and then we can watch how badly the server  degrades as it starts to thrash under the increased hostile client workload.

 

Getting the client mix right

 

You really do have to know your customer here. We always see all sorts of odd  behaviors from various corner case clients, which I have documented in other  posts here.

 

A full range of clients found on the BMC network would look like this;

  • AIX

  • DG-UX

  • Dynix/PTX (Sequent)

  • FreeBSD

  • HP-UX

  • IRIX

  • Linux - Fedora Core, Redhat AS, SuSE

  • NCR MP-RAS

  • Novell

  • OpenVMS - Multilan TCPIP

  • OS/2

  • OS X "Darwin"

  • SCO OpenServer

  • SCO Unixware

  • SINIX/Reliant Unix

  • Solaris

  • Tru64

  • Windows NT Workstation

  • Windows NT Server

  • Windows NT Enterprise Server

  • Windows 2000 Advanced Server

  • Windows 2000 Data Center

  • Windows 2000 Server

  • Windows 2000 Professional

  • Windows 2003 Server Standard Edition

  • Windows 2003 Server Enterprise Edition

  • Windows 2003 Server Enterprise Edition 64 bit

  • Windows 2003 Server Web Edition

  • Windows XP Professional

 

And there are of course sublevels of client, such as AIX 3.2.5 32 bit versus  AIX 5.3 64 bit and all the varieties in between. It will be fun collecting your  set, but if you don't, sooner or later the corner case will "byte" you.