Benchmarking bcache

Evaluating if bcache can/should be used on our compute nodes. Benchmarking was performed with iozone (revision 3.397) with the command line iozone -r 32 -s 1048576 -i 0 -i 1 -i 2 -i 8 -O -I -t 4 as 32kB record size looks like a representative one. As the machines are quad core machines, we use 4 parallel threads and perform tests with O_DIRECT in async fashion. To eliminate RAM caching we use mem=512M as a kernel boot option.

Summary (average number of IOPS per thread and workload)

type HDD SSD bcache bcache writeback
initial writer 648.77 1166.57 616.25 545.60
rewriter 619.52 1139.51 586.93 479.31
reader 86.72 2546.75 34.65 1672.49
re-reader 88.95 2549.07 37.62 1667.63
random reader 38.59 2481.68 38.57 1841.83
mixed workload 51.11 981.93 90.52 610.79
random writer 69.31 1103.15 60.05 357.18

Raw results

HDD only

        Iozone: Performance Test of File I/O
                Version $Revision: 3.397 $
                Compiled for 64 bit mode.
                Build: linux-AMD64 

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer.
                     Ben England.

        Run began: Thu Mar 27 16:33:55 2014

        Record Size 32 KB
        File size set to 1048576 KB
        OPS Mode. Output is in operations per second.
        O_DIRECT feature enabled
        Command line used: iozone -r 32 -s 1048576 -i 0 -i 1 -i 2 -i 8 -O -I -t 4
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
        Throughput test with 4 processes
        Each process writes a 1048576 Kbyte file in 32 Kbyte records

        Children see throughput for  4 initial writers  =    2595.10 ops/sec
        Parent sees throughput for  4 initial writers   =    2565.82 ops/sec
        Min throughput per process                      =     648.77 ops/sec 
        Max throughput per process                      =     648.79 ops/sec
        Avg throughput per process                      =     648.77 ops/sec
        Min xfer                                        =   32767.00 ops

        Children see throughput for  4 rewriters        =    2478.06 ops/sec
        Parent sees throughput for  4 rewriters         =    2454.33 ops/sec
        Min throughput per process                      =     619.50 ops/sec 
        Max throughput per process                      =     619.52 ops/sec
        Avg throughput per process                      =     619.52 ops/sec
        Min xfer                                        =   32767.00 ops

        Children see throughput for  4 readers          =     346.86 ops/sec
        Parent sees throughput for  4 readers           =     346.68 ops/sec
        Min throughput per process                      =      65.19 ops/sec 
        Max throughput per process                      =     123.47 ops/sec
        Avg throughput per process                      =      86.72 ops/sec
        Min xfer                                        =   17304.00 ops

        Children see throughput for 4 re-readers        =     355.80 ops/sec
        Parent sees throughput for 4 re-readers         =     355.67 ops/sec
        Min throughput per process                      =      71.76 ops/sec 
        Max throughput per process                      =     125.71 ops/sec
        Avg throughput per process                      =      88.95 ops/sec
        Min xfer                                        =   18708.00 ops

        Children see throughput for 4 random readers    =     154.37 ops/sec
        Parent sees throughput for 4 random readers     =     154.33 ops/sec
        Min throughput per process                      =       7.58 ops/sec 
        Max throughput per process                      =      98.02 ops/sec
        Avg throughput per process                      =      38.59 ops/sec
        Min xfer                                        =    2535.00 ops

        Children see throughput for 4 mixed workload    =     204.43 ops/sec
        Parent sees throughput for 4 mixed workload     =     204.02 ops/sec
        Min throughput per process                      =      11.66 ops/sec 
        Max throughput per process                      =      89.93 ops/sec
        Avg throughput per process                      =      51.11 ops/sec
        Min xfer                                        =    4249.00 ops

        Children see throughput for 4 random writers    =     277.22 ops/sec
        Parent sees throughput for 4 random writers     =     276.77 ops/sec
        Min throughput per process                      =      69.30 ops/sec 
        Max throughput per process                      =      69.31 ops/sec
        Avg throughput per process                      =      69.31 ops/sec
        Min xfer                                        =   32761.00 ops

SSD

        Iozone: Performance Test of File I/O
                Version $Revision: 3.397 $
                Compiled for 64 bit mode.
                Build: linux-AMD64 

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer.
                     Ben England.

        Run began: Thu Mar 27 16:29:44 2014

        Record Size 32 KB
        File size set to 1048576 KB
        OPS Mode. Output is in operations per second.
        O_DIRECT feature enabled
        Command line used: iozone -r 32 -s 1048576 -i 0 -i 1 -i 2 -i 8 -O -I -t 4
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
        Throughput test with 4 processes
        Each process writes a 1048576 Kbyte file in 32 Kbyte records

        Children see throughput for  4 initial writers  =    4666.29 ops/sec
        Parent sees throughput for  4 initial writers   =    4299.21 ops/sec
        Min throughput per process                      =    1073.92 ops/sec 
        Max throughput per process                      =    1267.17 ops/sec
        Avg throughput per process                      =    1166.57 ops/sec
        Min xfer                                        =   27771.00 ops

        Children see throughput for  4 rewriters        =    4558.05 ops/sec
        Parent sees throughput for  4 rewriters         =    4555.59 ops/sec
        Min throughput per process                      =    1135.01 ops/sec 
        Max throughput per process                      =    1143.81 ops/sec
        Avg throughput per process                      =    1139.51 ops/sec
        Min xfer                                        =   32516.00 ops

        Children see throughput for  4 readers          =   10187.00 ops/sec
        Parent sees throughput for  4 readers           =   10185.98 ops/sec
        Min throughput per process                      =    2530.82 ops/sec 
        Max throughput per process                      =    2557.96 ops/sec
        Avg throughput per process                      =    2546.75 ops/sec
        Min xfer                                        =   32421.00 ops

        Children see throughput for 4 re-readers        =   10196.26 ops/sec
        Parent sees throughput for 4 re-readers         =   10195.77 ops/sec
        Min throughput per process                      =    2537.48 ops/sec 
        Max throughput per process                      =    2570.97 ops/sec
        Avg throughput per process                      =    2549.07 ops/sec
        Min xfer                                        =   32342.00 ops

        Children see throughput for 4 random readers    =    9926.73 ops/sec
        Parent sees throughput for 4 random readers     =    9926.29 ops/sec
        Min throughput per process                      =    2459.02 ops/sec 
        Max throughput per process                      =    2511.85 ops/sec
        Avg throughput per process                      =    2481.68 ops/sec
        Min xfer                                        =   32079.00 ops

        Children see throughput for 4 mixed workload    =    3927.72 ops/sec
        Parent sees throughput for 4 mixed workload     =    3913.55 ops/sec
        Min throughput per process                      =     208.71 ops/sec 
        Max throughput per process                      =    1757.69 ops/sec
        Avg throughput per process                      =     981.93 ops/sec
        Min xfer                                        =    3891.00 ops

        Children see throughput for 4 random writers    =    4412.60 ops/sec
        Parent sees throughput for 4 random writers     =    4390.54 ops/sec
        Min throughput per process                      =    1097.77 ops/sec 
        Max throughput per process                      =    1107.07 ops/sec
        Avg throughput per process                      =    1103.15 ops/sec
        Min xfer                                        =   32493.00 ops



iozone test complete.

bcache (default)

        Iozone: Performance Test of File I/O
                Version $Revision: 3.397 $
                Compiled for 64 bit mode.
                Build: linux-AMD64 

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer.
                     Ben England.

        Run began: Thu Mar 27 17:07:45 2014

        Record Size 32 KB
        File size set to 1048576 KB
        OPS Mode. Output is in operations per second.
        O_DIRECT feature enabled
        Command line used: iozone -r 32 -s 1048576 -i 0 -i 1 -i 2 -i 8 -O -I -t 4
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
        Throughput test with 4 processes
        Each process writes a 1048576 Kbyte file in 32 Kbyte records

        Children see throughput for  4 initial writers  =    2465.01 ops/sec
        Parent sees throughput for  4 initial writers   =    2427.73 ops/sec
        Min throughput per process                      =     615.18 ops/sec 
        Max throughput per process                      =     617.12 ops/sec
        Avg throughput per process                      =     616.25 ops/sec
        Min xfer                                        =   32665.00 ops

        Children see throughput for  4 rewriters        =    2347.74 ops/sec
        Parent sees throughput for  4 rewriters         =    2320.40 ops/sec
        Min throughput per process                      =     574.66 ops/sec 
        Max throughput per process                      =     601.04 ops/sec
        Avg throughput per process                      =     586.93 ops/sec
        Min xfer                                        =   31330.00 ops

        Children see throughput for  4 readers          =     138.62 ops/sec
        Parent sees throughput for  4 readers           =     138.59 ops/sec
        Min throughput per process                      =      25.01 ops/sec 
        Max throughput per process                      =      47.44 ops/sec
        Avg throughput per process                      =      34.65 ops/sec
        Min xfer                                        =   17277.00 ops

        Children see throughput for 4 re-readers        =     150.46 ops/sec
        Parent sees throughput for 4 re-readers         =     150.44 ops/sec
        Min throughput per process                      =      26.45 ops/sec 
        Max throughput per process                      =      47.85 ops/sec
        Avg throughput per process                      =      37.62 ops/sec
        Min xfer                                        =   18117.00 ops

        Children see throughput for 4 random readers    =     154.27 ops/sec
        Parent sees throughput for 4 random readers     =     154.25 ops/sec
        Min throughput per process                      =      22.60 ops/sec 
        Max throughput per process                      =      49.57 ops/sec
        Avg throughput per process                      =      38.57 ops/sec
        Min xfer                                        =   14939.00 ops

        Children see throughput for 4 mixed workload    =     362.08 ops/sec
        Parent sees throughput for 4 mixed workload     =     361.10 ops/sec
        Min throughput per process                      =      36.68 ops/sec 
        Max throughput per process                      =     138.96 ops/sec
        Avg throughput per process                      =      90.52 ops/sec
        Min xfer                                        =    8670.00 ops

        Children see throughput for 4 random writers    =     240.20 ops/sec
        Parent sees throughput for 4 random writers     =     239.12 ops/sec
        Min throughput per process                      =      59.98 ops/sec 
        Max throughput per process                      =      60.24 ops/sec
        Avg throughput per process                      =      60.05 ops/sec
        Min xfer                                        =   32624.00 ops



iozone test complete.

bcache writeback cache

        Iozone: Performance Test of File I/O
                Version $Revision: 3.397 $
                Compiled for 64 bit mode.
                Build: linux-AMD64 

        Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins
                     Al Slater, Scott Rhine, Mike Wisner, Ken Goss
                     Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR,
                     Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner,
                     Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone,
                     Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root,
                     Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer.
                     Ben England.

        Run began: Thu Mar 27 18:05:01 2014

        Record Size 32 KB
        File size set to 1048576 KB
        OPS Mode. Output is in operations per second.
        O_DIRECT feature enabled
        Command line used: iozone -r 32 -s 1048576 -i 0 -i 1 -i 2 -i 8 -O -I -t 4
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 Kbytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
        Throughput test with 4 processes
        Each process writes a 1048576 Kbyte file in 32 Kbyte records

        Children see throughput for  4 initial writers  =    2182.38 ops/sec
        Parent sees throughput for  4 initial writers   =    2169.55 ops/sec
        Min throughput per process                      =     541.76 ops/sec 
        Max throughput per process                      =     548.65 ops/sec
        Avg throughput per process                      =     545.60 ops/sec
        Min xfer                                        =   32357.00 ops

        Children see throughput for  4 rewriters        =    1917.25 ops/sec
        Parent sees throughput for  4 rewriters         =    1916.55 ops/sec
        Min throughput per process                      =     477.48 ops/sec 
        Max throughput per process                      =     480.88 ops/sec
        Avg throughput per process                      =     479.31 ops/sec
        Min xfer                                        =   32538.00 ops

        Children see throughput for  4 readers          =    6689.95 ops/sec
        Parent sees throughput for  4 readers           =    6650.93 ops/sec
        Min throughput per process                      =    1635.10 ops/sec 
        Max throughput per process                      =    1778.46 ops/sec
        Avg throughput per process                      =    1672.49 ops/sec
        Min xfer                                        =   30127.00 ops

        Children see throughput for 4 re-readers        =    6670.52 ops/sec
        Parent sees throughput for 4 re-readers         =    6631.52 ops/sec
        Min throughput per process                      =    1632.89 ops/sec 
        Max throughput per process                      =    1768.31 ops/sec
        Avg throughput per process                      =    1667.63 ops/sec
        Min xfer                                        =   30259.00 ops

        Children see throughput for 4 random readers    =    7367.33 ops/sec
        Parent sees throughput for 4 random readers     =    7318.86 ops/sec
        Min throughput per process                      =    1779.25 ops/sec 
        Max throughput per process                      =    2021.97 ops/sec
        Avg throughput per process                      =    1841.83 ops/sec
        Min xfer                                        =   28835.00 ops

        Children see throughput for 4 mixed workload    =    2443.17 ops/sec
        Parent sees throughput for 4 mixed workload     =    2251.08 ops/sec
        Min throughput per process                      =     552.74 ops/sec 
        Max throughput per process                      =     637.42 ops/sec
        Avg throughput per process                      =     610.79 ops/sec
        Min xfer                                        =   28415.00 ops

        Children see throughput for 4 random writers    =    1428.71 ops/sec
        Parent sees throughput for 4 random writers     =    1415.27 ops/sec
        Min throughput per process                      =     355.79 ops/sec 
        Max throughput per process                      =     359.59 ops/sec
        Avg throughput per process                      =     357.18 ops/sec
        Min xfer                                        =   32422.00 ops



iozone test complete.
-- CarstenAulbert - 28 Mar 2014
Topic revision: r2 - 28 Mar 2014, CarstenAulbert
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback