Performance problems with X4500 using ZFS with NFSv3

When moving a large number of small files onto our X4500 boxes we found very bad performance numbers:

Job	local (UFS->ZFS)	network (xfs-NFS->ZFS) default mode	network (xfs-NFS->ZFS) no cache flushing
copying single 5GB file	6.5s	51.5s	50.9s
removing this	0.0s	0.5s	0.6s
copying 5000 3 kB files	0.3s	257s	16.9s
removing these	0.1s	42.6s	5.2s

(note: This was done via single 1 Gb/s link, thus copying the large file was about line speed)

-- CarstenAulbert - 12 Jun 2008