Benchmarking distributed file systems

stupid fast tests first, all using small compute nodes and use iozone -r 32 -s $((2**24+2**25)) -i 0 -i 1 -i 2 -i 8 -O -I

Single iozone thread

local SSD test

a9004 with /local/ssd

iozone -r 32 -s $((2**24+2**25)) -i 0 -i 1 -i 2 -i 8 -O -I
 [...]
                                                      random  random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
        50331648      32    1466    4644    12709    12561    4286     564

Quobyte

Use a9001/2/3 as storage servers and a9004 as a networked client

Iozone: Performance Test of File I/O
[...]
                                                            random  random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
        50331648      32     201     248      736      753     788     339

BeeGFS

Using a9005 as config, a9006 as meta data, a9007...a9009 as storage and a9010 as client:

[...]
     
                                                            random  random    bkwd   record   stride
              KB  reclen   write rewrite    read    reread    read   write    read  rewrite     read   fwrite frewrite   fread  freread
        50331648      32    1025    1047     1031     1033     935    1022

Comparison

This table summarizes the single thread performance, all values in ops/s according to above definition.

Test	local SSD	Quobyte	!BeeGFS
write	1466	201	1025
rewrite	4644	248	1047
read	12709	736	1031
reread	12561	753	1033
random read	4286	788	935
random write	564	339	1022

Massively parallel test

Here, we run with 15 parallel threads using iozone -r 32 -s 4194304 -i 0 -i 1 -i 2 -i 8 -O -I -t 15. For comparison, we will only concentrate on the throughput the parent sees ignoring potential problems with min/avg/max for now. On the local system, the tests are run twice, once with 15 threads and once with 5 threads but thrice the size.

All results are again in ops/s.

Test	local SSD 5 threads	local SSD 15 threads	Quobyte	BeeGFS single 1Gb/s client	BeeGFS 3 clients (summed)	BeeGFS single 10Gb/s client
Initial write	3861	3703	796	3718	7648	6901
rewrite	4556	4469	1182	3745	6076	7905
reader	10331	10523	1841	3771	11289	10871
re-read	10296	10519	1860	3771	11296	10918
random read	10128	10479	3682	3771	10888	9895
mixed workload	3917	3981	1584	6346	8260	7914
random write	4286	4196	1538	3721	6074	7097

Note for BeeGFS single client test: Most of the test was limited by the 1 Gb/s NIC of a9010, hence the constant results in most categories!

-- CarstenAulbert - 21 Aug 2014