Storage server
- Huawei 5288 v5
- 36 x 14 HDDs
- 192 GB RAM
- Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz
sizes
set |
size (TiB) |
9 x ( 3 + 1 ) |
322.576 |
18 x ( 1 + 1 ) |
221.783 |
6 x ( 4 + 2 ) |
295.725 |
6 x ( 5 + 1 ) |
354.696 |
4 x ( 7 + 2 ) |
338.242 |
3 x ( 10 + 2 ) |
338.253 |
1 x ( 33 + 3 ) |
394.629 |
12 x ( 2 + 1 ) |
295.664 |
- arccache 64G (previous test show 4GB and more yield same performance)
- see test suite
randomread (IO/s)
zpool setup |
12 files |
24 |
48 |
96 |
192 |
384 |
768 |
12x2p1 |
1034.55+-46.04 |
1037.91+-45.94 |
1040.16+-45.27 |
1036.82+-46.59 |
1040.31+-46.87 |
1039.04+-46.70 |
925.14+-116.59 |
18x1p1 |
1487.45+-37.86 |
1487.07+-39.55 |
1480.71+-37.44 |
1459.17+-45.87 |
1444.80+-48.36 |
1443.39+-52.37 |
1332.57+-123.04 |
1x33p3 |
217.18+-38.65 |
220.80+-38.03 |
218.80+-35.53 |
219.27+-34.35 |
220.03+-32.49 |
218.92+-34.79 |
174.29+-33.01 |
3x10p2 |
482.41+-57.58 |
486.35+-55.83 |
486.32+-56.37 |
484.06+-55.47 |
484.56+-56.00 |
482.27+-54.53 |
361.16+-73.55 |
4x7p2 |
586.10+-53.44 |
584.92+-51.90 |
584.67+-50.83 |
586.86+-49.41 |
586.27+-51.42 |
585.80+-51.82 |
460.52+-83.13 |
6x4p2 |
793.73+-50.74 |
756.89+-57.26 |
760.46+-58.62 |
758.14+-58.83 |
755.63+-57.32 |
758.87+-57.45 |
633.92+-105.01 |
6x5p1 |
709.75+-57.43 |
708.84+-55.15 |
708.78+-57.53 |
710.51+-59.53 |
706.78+-57.00 |
711.59+-59.57 |
580.94+-100.29 |
9x3p1 |
894.94+-58.67 |
879.84+-63.84 |
881.65+-58.35 |
880.28+-62.46 |
883.58+-59.32 |
881.67+-60.36 |
755.65+-116.57 |
streamread
zpool setup |
12 files |
24 |
48 |
96 |
192 |
384 |
768 |
12x2p1 |
2.50+-0.24 GB/s |
2.49+-0.24 GB/s |
2.47+-0.24 GB/s |
2.48+-0.25 GB/s |
2.45+-0.24 GB/s |
2.48+-0.25 GB/s |
2.08+-0.22 GB/s |
18x1p1 |
2.63+-0.25 GB/s |
2.65+-0.24 GB/s |
2.67+-0.24 GB/s |
2.65+-0.24 GB/s |
2.67+-0.25 GB/s |
2.67+-0.25 GB/s |
2.56+-0.44 GB/s |
1x33p3 |
2.22+-0.09 GB/s |
2.22+-0.09 GB/s |
2.21+-0.09 GB/s |
2.22+-0.09 GB/s |
2.21+-0.09 GB/s |
2.21+-0.10 GB/s |
1.24+-0.24 GB/s |
3x10p2 |
2.39+-0.27 GB/s |
2.35+-0.27 GB/s |
2.34+-0.26 GB/s |
2.34+-0.27 GB/s |
2.34+-0.27 GB/s |
2.34+-0.27 GB/s |
1.37+-0.22 GB/s |
4x7p2 |
2.43+-0.26 GB/s |
2.42+-0.26 GB/s |
2.41+-0.26 GB/s |
2.41+-0.26 GB/s |
2.40+-0.26 GB/s |
2.40+-0.26 GB/s |
1.49+-0.22 GB/s |
6x4p2 |
2.98+-0.39 GB/s |
3.01+-0.35 GB/s |
2.99+-0.37 GB/s |
3.00+-0.37 GB/s |
3.05+-0.34 GB/s |
3.01+-0.38 GB/s |
2.41+-0.26 GB/s |
6x5p1 |
2.58+-0.30 GB/s |
2.58+-0.29 GB/s |
2.58+-0.29 GB/s |
2.56+-0.28 GB/s |
2.56+-0.27 GB/s |
2.56+-0.27 GB/s |
1.79+-0.23 GB/s |
9x3p1 |
2.47+-0.30 GB/s |
2.50+-0.29 GB/s |
2.51+-0.28 GB/s |
2.52+-0.28 GB/s |
2.53+-0.28 GB/s |
2.54+-0.27 GB/s |
1.79+-0.19 GB/s |
streamwrite
zpool setup |
12 files |
24 |
48 |
96 |
192 |
384 |
768 |
1536 |
3072 |
6144 |
12x2p1 |
2.60+-0.13 GB/s |
2.46+-0.18 GB/s |
2.55+-0.12 GB/s |
2.52+-0.12 GB/s |
2.50+-0.11 GB/s |
2.49+-0.16 GB/s |
2.52+-0.12 GB/s |
2.52+-0.13 GB/s |
2.52+-0.12 GB/s |
2.10+-0.14 GB/s |
18x1p1 |
2.11+-0.11 GB/s |
2.07+-0.11 GB/s |
2.06+-0.11 GB/s |
2.04+-0.13 GB/s |
2.05+-0.11 GB/s |
2.04+-0.12 GB/s |
2.05+-0.11 GB/s |
2.02+-0.12 GB/s |
2.04+-0.13 GB/s |
1.79+-0.10 GB/s |
1x33p3 |
1.48+-0.06 GB/s |
1.47+-0.06 GB/s |
1.47+-0.06 GB/s |
1.47+-0.06 GB/s |
1.46+-0.07 GB/s |
1.45+-0.07 GB/s |
1.45+-0.07 GB/s |
1.44+-0.09 GB/s |
1.46+-0.06 GB/s |
1.20+-0.06 GB/s |
3x10p2 |
2.76+-0.13 GB/s |
2.71+-0.14 GB/s |
2.73+-0.15 GB/s |
2.72+-0.14 GB/s |
2.70+-0.13 GB/s |
2.70+-0.15 GB/s |
2.69+-0.14 GB/s |
2.70+-0.15 GB/s |
2.68+-0.13 GB/s |
2.13+-0.18 GB/s |
4x7p2 |
2.79+-0.16 GB/s |
2.73+-0.16 GB/s |
2.76+-0.12 GB/s |
2.71+-0.15 GB/s |
2.75+-0.16 GB/s |
2.72+-0.16 GB/s |
2.72+-0.16 GB/s |
2.73+-0.14 GB/s |
2.67+-0.15 GB/s |
2.16+-0.17 GB/s |
6x4p2 |
2.59+-0.11 GB/s |
2.48+-0.09 GB/s |
2.55+-0.10 GB/s |
2.51+-0.07 GB/s |
2.52+-0.11 GB/s |
2.50+-0.17 GB/s |
2.48+-0.13 GB/s |
2.48+-0.11 GB/s |
2.46+-0.14 GB/s |
2.02+-0.13 GB/s |
6x5p1 |
2.94+-0.11 GB/s |
2.84+-0.16 GB/s |
2.85+-0.14 GB/s |
2.86+-0.16 GB/s |
2.85+-0.14 GB/s |
2.88+-0.14 GB/s |
2.82+-0.17 GB/s |
2.88+-0.15 GB/s |
2.82+-0.19 GB/s |
2.25+-0.10 GB/s |
9x3p1 |
2.75+-0.13 GB/s |
2.66+-0.13 GB/s |
2.68+-0.15 GB/s |
2.67+-0.14 GB/s |
2.66+-0.13 GB/s |
2.66+-0.14 GB/s |
2.69+-0.15 GB/s |
2.65+-0.14 GB/s |
2.67+-0.14 GB/s |
2.17+-0.14 GB/s |
resilver times
- create 10 x 8GB files with random content in shared memory
mkdir -p /dev/shm/data
for i in $(seq -w 10); do dd if=<(openssl enc -aes-256-ctr -pass pass:"$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64)" -nosalt < /dev/zero) of=/dev/shm/data/random.$i bs=1M count=8192 iflag=fullblock; done
After /tank has been filled to 100% /dev/sdb has been taken offline and filled partially with zeros.
- zpool offline tank /dev/sdb
- dd if=/dev/zero of=/dev/sdb bs=1G count=10
- zpool replace tank -f /dev/sdb /dev/sdb
- zpool scrub tank
durations
resilver times under load
read
The pool contains only one read set.
The entire pool has been filled with 8GiB files as described above and subsequently read in with dd.
dd if=$i of=/dev/null bs=1M status=noxfer.
The times have been measured per file before, during and after resilvering.
write
Write constant in 12 files in parallel using fio
; stream write
[global]
name=fio-write
readwrite=write
bs=128K
numjobs=12
time_based=0
runtime=300
directory=/tank/fio
log_avg_msec=5000
log_hist_msec=5000
stats=1
[create]
create_on_open=1
unlink=1
nrfiles=1
filename_format=fio-write-$jobnum-$filenum
filesize=1G
ioengine=libaio
file_append=0
write_iops_log=/root/fio/run/logs
write_bw_log=/root/fio/run/logs
write_lat_log=/root/fio/run/logs
write_hist_log=/root/fio/run/logs
+++ similar consideration but only writing one file at a time:
--
HenningFehrmann - 18 Dec 2020