NFS Performance
As many of you, i did start to work with NFS when performance was not a problem. Right? Well, more or less… performance was not the first problem. The first real problem was get the whole thing working! Compatibility was a huge problem, and i’m not talking about GNU/Linux and Unix compatibility, that was some kind of art to get things working. But i’m talking about GNU/Linux and GNU/Linux. The same distro, with the same version was a problem, imagine different distros!
So, after that, in general after remove all the NFS stuff from the distros and installing from source the nfs-utils, rpc, and etc (homogeneous install), everything working… ok, now performance was the problem. ;-)
Well, welcome back to 2009! Compatibility is a problem? No, NFSv3 is a established standard and i have no experiences with incompatibility theses days. And performance? I admit that i did think that was another problem that we did leave behind too. But i’m wrong. Yeah, i know if we want to deploy a high performance environment, or we need to get *more* than a normal configuration would give us, we need to do some dirty work. But the performance numbers i do see on GNU/Linux RedHat clients were some crazy ones!
I did a simple test like:
iozone -s1g -r8 -i 0 -i 2 -t 1 -c -o -O
And the performance for random writes were 1500 ops! Changing the threads to: 2, 3, 6 and 12 did not change *nothing*. The numbers were 1200, 1600, 1500…
A OpenSolaris client got much better numbers, like 1500, 2500, 4000, 5000! The same as the local ZFS performance/NFS server (with one intel SSD). The difference was that on the local NFS server, i got 3300 ops with a single thread.
At least the Ubuntu client was better, with numbers not so good like the OpenSolaris client (2009.06), but way better than the RedHat!
So, if Brendan Gregg or Roch see this post by accident, i think you will need to do something like NetApp GNU/Linux patches. Brendan is working in getting on the fishworks limits (I think the clients they are using on the tests are just OpenSolaris/Solaris). But as a enterprise GNU/Linux distro (RedHat), present a lot on the datacenters, i fear the Storage 7000 series from SUN/Oracle be like a ferrari without road to run. I know that thinking quickly we can say: “That’s not my proble…”, but indirectly can be.
peace
I did forget to mention something and let me put it here like a comment:
1 – The servers were almost the same (the ethernet nic was a broadcom on RH).
2 – The tests were just one after another on the same machine (adding threads).
3 – The above sequence did run two times, so the numbers were confirmed the same.
4 – If there is some cache between the executions (t1, t2, t3, etc), ok let’s see the numbers.
5 – Doing the same test changing the RH kernel for a vanila one, the numbers were the same as Ubuntu GNU/Linux. Yet using the broadcom interface.
I know in the RH (standard) case we can have problems in any level on the stack, but the whole picture was really bad.