You are here: Foswiki>ATLAS Web>EatHDiskConfigTests (07 Apr 2016, CarstenAulbert)Edit Attach

Simple E@H disk layout tests

Summary

Table with timing results (time in seconds, lower is better):

setup md3log z2log/cache zfs3log/cache btrfs-raid10 xfs10logdevAG xfs10logdev xfs10 zfs2mirror/log/cache
create fan-out 7.1 1.4 1.4 0.7 153.0 1.9 3.4 1.3
O1 tar 77.3 42.2 45.4 41.1 73.0 72.9 76.5 52.9
O1 mv 9.5 2.3 2.2 1.1 21.4 4.0 6.6 2.1
S5GC1 tar 51.7 41.6 50.4 40.0 76.5 37.9 55.7 46.6
S5GC1 mv 23.5 6.2 6.4 2.6 35.1 9.0 13.5 6.3
S6BuFU tar 114.4 292.6 368.1 99.9 105.9 107.2 147.1 328.0
S6BuFU mv 38.6 13.3 13.7 6.3 51.5 24.8 32.1 13.9
S6BuLVE tar 68.5 111.0 106.2 50.6 85.1 61.5 70.8 109.1
S6BuLVE mv 34.1 21.2 24.9 6.4 48.2 31.7 30.7 23.2
S6CasA tar 47.7 37.1 39.2 30.3 44.6 41.1 43.4 36.5
S6CasA mv 32.5 27.4 31.9 7.9 45.0 22.6 26.5 28.1
hardlink A->B ? ? ? ? ? ? ? 60.5
tarball create 1486.6 5240.6 2439.7 2182.3 1701.5 1600.6 1585.1 2452.4
remove 209.9 3394.8 3437.1 95.7 380.9 207.6 284.7 3444.5
untar all 698.5 214.5 324.3 194.2 543.7(?) 639.7 650.3 243.3

Raw results

MD based three disk mirrors with logdev on NVME (md3log) 

Create 26 fan out dirs: Runtime: 7.1s
/srv/O1AS20-100I_canonical_results_2016_03_30.tar - 17882 MiB - 23488 files:
     tar: Runtime: 30.1s
     umount: Runtime: 47.2s
Moving 23488 files into fan-out:
     mv: Runtime: 8.2s
     umount: Runtime: 1.3s
/srv/S5GC1HF_canonical_results_2014_07_15.tar - 12665 MiB - 66313 files:
     tar: Runtime: 32.0s
     umount: Runtime: 19.7s
Moving 66313 files into fan-out:
     mv: Runtime: 21.0s
     umount: Runtime: 2.5s
/srv/S6BucketFU3UB_canonical_results_2015_09_15.tar - 34313 MiB - 178221 files:
     tar: Runtime: 92.8s
     umount: Runtime: 21.6s
Moving 178221 files into fan-out:
     mv: Runtime: 31.5s
     umount: Runtime: 7.1s
/srv/S6BucketLVE_canonical_results_2013_02_06.tar - 16580 MiB - 122703 files:
     tar: Runtime: 42.3s
     umount: Runtime: 26.2s
Moving 122703 files into fan-out:
     mv: Runtime: 31.4s
     umount: Runtime: 2.7s
/srv/S6CasA_canonical_results_2014_01_21.tar - 10245 MiB - 74347 files:
     tar: Runtime: 28.3s
     umount: Runtime: 19.4s
Moving 74347 files into fan-out:
     mv: Runtime: 27.6s
     umount: Runtime: 4.9s
Creating 5 tarballs in-place of fanned-out files in parallel: Runtime: 1486.6s
Remove everything: Runtime: 209.9s
Untar everything again: Runtime: 698.5s

ZFS raidz2 with log and cache (z2log/cache) 

Create 26 fan out dirs: Runtime: 1.4s
/srv/O1AS20-100I_canonical_results_2016_03_30.tar - 17882 MiB - 23488 files:
     tar: Runtime: 40.5s
     umount: Runtime: 1.7s
Moving 23488 files into fan-out:
     mv: Runtime: 1.3s
     umount: Runtime: 1.0s
/srv/S5GC1HF_canonical_results_2014_07_15.tar - 12665 MiB - 66313 files:
     tar: Runtime: 39.6s
     umount: Runtime: 2.0s
Moving 66313 files into fan-out:
     mv: Runtime: 5.0s
     umount: Runtime: 1.2s
/srv/S6BucketFU3UB_canonical_results_2015_09_15.tar - 34313 MiB - 178221 files:
     tar: Runtime: 289.6s
     umount: Runtime: 3.0s
Moving 178221 files into fan-out:
     mv: Runtime: 11.5s
     umount: Runtime: 1.8s
/srv/S6BucketLVE_canonical_results_2013_02_06.tar - 16580 MiB - 122703 files:
     tar: Runtime: 108.4s
     umount: Runtime: 2.6s
Moving 122703 files into fan-out:
     mv: Runtime: 19.6s
     umount: Runtime: 1.6s
/srv/S6CasA_canonical_results_2014_01_21.tar - 10245 MiB - 74347 files:
     tar: Runtime: 34.8s
     umount: Runtime: 2.3s
Moving 74347 files into fan-out:
     mv: Runtime: 26.2s
     umount: Runtime: 1.2s
Creating 5 tarballs in-place of fanned-out files in parallel: Runtime: 5240.6s
Remove everything: Runtime: 3394.8s
Untar everything again: 214.5s

ZFS with 3 disk mirrors and log/cache (zfs3log/cache) 

Create 26 fan out dirs: Runtime: 1.4s
/srv/O1AS20-100I_canonical_results_2016_03_30.tar - 17882 MiB - 23488 files:
     tar: Runtime: 43.5s
     umount: Runtime: 1.9s
Moving 23488 files into fan-out:
     mv: Runtime: 1.4s
     umount: Runtime: 0.8s
/srv/S5GC1HF_canonical_results_2014_07_15.tar - 12665 MiB - 66313 files:
     tar: Runtime: 48.7s
     umount: Runtime: 1.7s
Moving 66313 files into fan-out:
     mv: Runtime: 5.6s
     umount: Runtime: 0.8s
/srv/S6BucketFU3UB_canonical_results_2015_09_15.tar - 34313 MiB - 178221 files:
     tar: Runtime: 365.0s
     umount: Runtime: 3.1s
Moving 178221 files into fan-out:
     mv: Runtime: 12.5s
     umount: Runtime: 1.2s
/srv/S6BucketLVE_canonical_results_2013_02_06.tar - 16580 MiB - 122703 files:
     tar: Runtime: 103.5s
     umount: Runtime: 2.7s
Moving 122703 files into fan-out:
     mv: Runtime: 23.8s
     umount: Runtime: 1.1s
/srv/S6CasA_canonical_results_2014_01_21.tar - 10245 MiB - 74347 files:
     tar: Runtime: 34.3s
     umount: Runtime: 4.9s
Moving 74347 files into fan-out:
     mv: Runtime: 30.9s
     umount: Runtime: 1.0s
Creating 5 tarballs in-place of fanned-out files in parallel: Runtime: 2439.7s
Remove everything: Runtime: 3437.1s
Untar everything again: Runtime: 324.3s

BTRFS (btrfs-raid10) 

Create 26 fan out dirs: Runtime: 0.6s
/srv/O1AS20-100I_canonical_results_2016_03_30.tar - 17882 MiB - 23488 files:
     tar: Runtime: 34.8s
     umount: Runtime: 6.3s
Moving 23488 files into fan-out:
     mv: Runtime: 0.7s
     umount: Runtime: 0.4s
/srv/S5GC1HF_canonical_results_2014_07_15.tar - 12665 MiB - 66313 files:
     tar: Runtime: 34.1s
     umount: Runtime: 5.9s
Moving 66313 files into fan-out:
     mv: Runtime: 1.8s
     umount: Runtime: 0.8s
/srv/S6BucketFU3UB_canonical_results_2015_09_15.tar - 34313 MiB - 178221 files:
     tar: Runtime: 90.7s
     umount: Runtime: 9.2s
Moving 178221 files into fan-out:
     mv: Runtime: 4.8s
     umount: Runtime: 1.5s
/srv/S6BucketLVE_canonical_results_2013_02_06.tar - 16580 MiB - 122703 files:
     tar: Runtime: 42.6s
     umount: Runtime: 8.0s
Moving 122703 files into fan-out:
     mv: Runtime: 5.1s
     umount: Runtime: 1.3s
/srv/S6CasA_canonical_results_2014_01_21.tar - 10245 MiB - 74347 files:
     tar: Runtime: 25.6s
     umount: Runtime: 4.7s
Moving 74347 files into fan-out:
     mv: Runtime: 6.5s
     umount: Runtime: 1.4s
Creating 5 tarballs in-place of fanned-out files in parallel: Runtime: 2182.3s
Remove everything: Runtime: 95.7s
Untar everything again: Runtime: 194.2s

MD RAID10 XFS with logdev (xfs10logdevAG)

Test with massive number of AG groups: 
agsize rounded to 1073152, swidth = 768
meta-data=/dev/md1               isize=512    agcount=8192, agsize=1073152 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1
data     =                       bsize=4096   blocks=8790587136, imaxpct=5
         =                       sunit=128    swidth=768 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =/dev/md10              bsize=4096   blocks=131072, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0



Create 26 fan out dirs: Runtime: 153.0s
/srv/O1AS20-100I_canonical_results_2016_03_30.tar - 17882 MiB - 23488 files:
     tar: Runtime: 36.1s
     umount: Runtime: 36.9s
Moving 23488 files into fan-out:
     mv: Runtime: 19.0s
     umount: Runtime: 2.4s
/srv/S5GC1HF_canonical_results_2014_07_15.tar - 12665 MiB - 66313 files:
     tar: Runtime: 32.3s
     umount: Runtime: 44.2s
Moving 66313 files into fan-out:
     mv: Runtime: 32.6s
     umount: Runtime: 2.5s
/srv/S6BucketFU3UB_canonical_results_2015_09_15.tar - 34313 MiB - 178221 files:
     tar: Runtime: 89.7s
     umount: Runtime: 16.2s
Moving 178221 files into fan-out:
     mv: Runtime: 48.1s
     umount: Runtime: 3.4s
/srv/S6BucketLVE_canonical_results_2013_02_06.tar - 16580 MiB - 122703 files:
     tar: Runtime: 60.6s
     umount: Runtime: 24.5s
Moving 122703 files into fan-out:
     mv: Runtime: 44.3s
     umount: Runtime: 3.9s
/srv/S6CasA_canonical_results_2014_01_21.tar - 10245 MiB - 74347 files:
     tar: Runtime: 25.4s
     umount: Runtime: 19.2s
Moving 74347 files into fan-out:
     mv: Runtime: 39.1s
     umount: Runtime: 5.9s
Creating 5 tarballs in-place of fanned-out files in parallel: Runtime: 1701.5s
Remove everything: Runtime: 380.9s
Untar everything again: tar: A/101: Cannot mkdir: No such file or directory
tar: A/1: Cannot mkdir: No such file or directory
tar: Exiting with failure status due to previous errors
tar: Exiting with failure status due to previous errors
Runtime: 543.7s

XFS RAID10 logdev (xfs10logdev) 

meta-data=/dev/md1               isize=512    agcount=33, agsize=268435328 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1
data     =                       bsize=4096   blocks=8790587136, imaxpct=5
         =                       sunit=128    swidth=768 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =/dev/md10              bsize=4096   blocks=131072, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0


Create 26 fan out dirs: Runtime: 1.9s
/srv/O1AS20-100I_canonical_results_2016_03_30.tar - 17882 MiB - 23488 files:
     tar: Runtime: 36.7s
     umount: Runtime: 36.2s
Moving 23488 files into fan-out:
     mv: Runtime: 2.9s
     umount: Runtime: 1.1s
/srv/S5GC1HF_canonical_results_2014_07_15.tar - 12665 MiB - 66313 files:
     tar: Runtime: 31.7s
     umount: Runtime: 16.2s
Moving 66313 files into fan-out:
     mv: Runtime: 8.0s
     umount: Runtime: 1.0s
/srv/S6BucketFU3UB_canonical_results_2015_09_15.tar - 34313 MiB - 178221 files:
     tar: Runtime: 85.5s
     umount: Runtime: 21.7s
Moving 178221 files into fan-out:
     mv: Runtime: 22.0s
     umount: Runtime: 2.8s
/srv/S6BucketLVE_canonical_results_2013_02_06.tar - 16580 MiB - 122703 files:
     tar: Runtime: 40.6s
     umount: Runtime: 20.9s
Moving 122703 files into fan-out:
     mv: Runtime: 29.6s
     umount: Runtime: 2.1s
/srv/S6CasA_canonical_results_2014_01_21.tar - 10245 MiB - 74347 files:
     tar: Runtime: 23.6s
     umount: Runtime: 17.5s
Moving 74347 files into fan-out:
     mv: Runtime: 21.6s
     umount: Runtime: 1.0s
Creating 5 tarballs in-place of fanned-out files in parallel: Runtime: 1600.6s
Remove everything: Runtime: 207.6s
Untar everything again: Runtime: 639.7s

XFS RAID10 std (xfs10) 

meta-data=/dev/md1               isize=512    agcount=33, agsize=268435328 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1
data     =                       bsize=4096   blocks=8790587136, imaxpct=5
         =                       sunit=128    swidth=768 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Create 26 fan out dirs: Runtime: 3.4s /srv/O1AS20-100I_canonical_results_2016_03_30.tar - 17882 MiB - 23488 files: tar: Runtime: 36.7s umount: Runtime: 39.8s Moving 23488 files into fan-out: mv: Runtime: 4.4s umount: Runtime: 2.2s /srv/S5GC1HF_canonical_results_2014_07_15.tar - 12665 MiB - 66313 files: tar: Runtime: 41.3s umount: Runtime: 14.4s Moving 66313 files into fan-out: mv: Runtime: 11.2s umount: Runtime: 2.3s /srv/S6BucketFU3UB_canonical_results_2015_09_15.tar - 34313 MiB - 178221 files: tar: Runtime: 85.2s umount: Runtime: 61.9s Moving 178221 files into fan-out: mv: Runtime: 11.5s umount: Runtime: 20.6s /srv/S6BucketLVE_canonical_results_2013_02_06.tar - 16580 MiB - 122703 files: tar: Runtime: 40.6s umount: Runtime: 30.2s Moving 122703 files into fan-out: mv: Runtime: 16.0s umount: Runtime: 14.7s /srv/S6CasA_canonical_results_2014_01_21.tar - 10245 MiB - 74347 files: tar: Runtime: 22.8s umount: Runtime: 20.5s Moving 74347 files into fan-out: mv: Runtime: 23.5s umount: Runtime: 3.0s Creating 5 tarballs in-place of fanned-out files in parallel: Runtime: 1585.1s Remove everything: Runtime: 284.7s Untar everything again: Runtime: 650.3s

ZFS with two disk mirrors and log/cache (zfs2mirror/log/cache) 

Create 26 fan out dirs: Runtime: 1.3s
/srv/O1AS20-100I_canonical_results_2016_03_30.tar - 17882 MiB - 23488 files:
     tar: Runtime: 51.5s
     umount: Runtime: 1.4s
Moving 23488 files into fan-out:
     mv: Runtime: 1.3s
     umount: Runtime: 0.8s
/srv/S5GC1HF_canonical_results_2014_07_15.tar - 12665 MiB - 66313 files:
     tar: Runtime: 44.9s
     umount: Runtime: 1.7s
Moving 66313 files into fan-out:
     mv: Runtime: 5.6s
     umount: Runtime: 0.7s
/srv/S6BucketFU3UB_canonical_results_2015_09_15.tar - 34313 MiB - 178221 files:
     tar: Runtime: 325.1s
     umount: Runtime: 2.9s
Moving 178221 files into fan-out:
     mv: Runtime: 12.6s
     umount: Runtime: 1.3s
/srv/S6BucketLVE_canonical_results_2013_02_06.tar - 16580 MiB - 122703 files:
     tar: Runtime: 106.5s
     umount: Runtime: 2.6s
Moving 122703 files into fan-out:
     mv: Runtime: 22.1s
     umount: Runtime: 1.1s
/srv/S6CasA_canonical_results_2014_01_21.tar - 10245 MiB - 74347 files:
     tar: Runtime: 34.4s
     umount: Runtime: 2.1s
Moving 74347 files into fan-out:
     mv: Runtime: 27.0s
     umount: Runtime: 1.1s
hard linking A to B: Runtime: 60.5s
Creating 5 tarballs in-place of fanned-out files in parallel: Runtime: 2452.4s
Remove everything: Runtime: 3444.5s
Untar everything again: Runtime: 243.3s

-- CarstenAulbert - 06 Apr 2016
Topic attachments
I Attachment Action Size Date Who Comment
bench.shsh bench.sh manage 2 K 07 Apr 2016 - 06:21 CarstenAulbert simple benchmark script
Edit  | Attach | Print version | History: r7 < r6 < r5 < r4 | Backlinks | View wiki text | Edit wiki text | More topic actions
Topic revision: r7 - 07 Apr 2016, CarstenAulbert
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback