You are here: Foswiki>ATLAS Web>X4500>X4500_zpoolconfiguration (18 Jun 2008, HenningFehrmann)Edit Attach

Possible zpool configurations

In the standard setup there are two system disks (either on controller c5 or c6 depending if one installs the box via a USB device - can this be fixed?) leaving 46 disks for storage. We have played around with a few possible set-ups:

raidz1

  c0 c1 c4 c5 c6 c7
t7 8 8 8 8 8 8
t6 7 7 7 7 7 7
t5 6 6 6 6 6 6
t4 5 5 5 S 5 5
t3 4 4 4 4 4 4
t2 3 3 3 3 3 3
t1 2 2 2 2 2 2
t0 1 1 1 S 1 1

Command to create this: 
zpool create -f $ZPOOL raidz1 c{0,1,4,6,7}t0d0 \
            raidz1 c{0,1,4,5,6,7}t1d0 \
            raidz1 c{0,1,4,5,6,7}t2d0 \
            raidz1 c{0,1,4,5,6,7}t3d0 \
            raidz1 c{0,1,4,6,7}t4d0 \
            raidz1 c{0,1,4,5,6,7}t5d0 \
            raidz1 c{0,1,4,5,6,7}t6d0 \
            raidz1 c{0,1,4,5,6,7}t7d0

results:dd if=/dev/zero of=/atlashome/zero/z bs=1024k count=100000 

zpool iostat 5
               capacity     operations    bandwidth
pool         used  avail   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
atlashome   2.30G  20.8T      0  3.09K      0   387M
atlashome   4.82G  20.8T      0  3.38K      0   424M
atlashome   7.29G  20.8T      0  3.33K      0   417M
atlashome   9.68G  20.8T      0  3.21K      0   402M
atlashome   12.0G  20.8T      0  3.18K      0   399M
atlashome   14.4G  20.8T      0  3.22K      0   403M
atlashome   17.4G  20.8T      0  3.98K      0   500M
atlashome   20.0G  20.8T      0  3.55K      0   445M
atlashome   23.1G  20.8T      0  4.14K      0   520M
atlashome   26.2G  20.8T      0  4.12K      0   518M
atlashome   29.3G  20.8T      0  4.19K      0   526M
atlashome   32.4G  20.8T      0  4.14K      0   519M
atlashome   35.2G  20.8T      0  3.79K      0   475M
atlashome   38.2G  20.8T      0  3.99K      0   501M
atlashome   41.4G  20.8T      0  4.32K      0   542M
atlashome   44.6G  20.8T      0  4.33K      0   544M
atlashome   47.7G  20.8T      0  4.11K      0   516M
atlashome   50.5G  20.8T      0  3.72K      0   467M
atlashome   53.4G  20.8T      0  3.92K      0   492M
atlashome   56.3G  20.8T      0  3.85K      0   483M
atlashome   59.6G  20.8T      0  4.48K      0   563M
atlashome   62.6G  20.8T      0  3.97K      0   498M
atlashome   65.3G  20.8T      0  3.64K      0   457M

This set-up yields no hot spare drives, good redundancy versus controller failure and a net storage of 6 * 5 + 2 * 4 = 38 drives or 19 TB with 500 GB drives.

raidz2

  c0 c1 c4 c5 c6 c7
t7 4 4 4 4 4 4
t6 4 4 4 4 H 4
t5 3 3 3 3 3 3
t4 3 3 3 S 3 3
t3 2 2 2 2 2 2
t2 2 H 2 2 2 2
t1 1 1 1 1 1 1
t0 1 1 1 S 1 1

Command to create this: 
zpool create -f $ZPOOL raidz2 c{0,1,4,6,7}t0d0 c{5,0,1,4,6,7}t1d0 \
           raidz2 c{5,0,1,6,7}t2d0 c{5,0,1,4,6,7}t3d0 \
           raidz2 c{0,1,4,6,7}t4d0 c{5,0,1,4,6,7}t5d0 \
           raidz2 c{5,0,1,4,7}t6d0 c{5,0,1,4,6,7}t7d0 \
           spare c1t2d0 c6t6d0

results:dd if=/dev/zero of=/atlashome/zero/z bs=1024k count=100000 

zpool  iostat 5
               capacity     operations    bandwidth
pool         used  avail   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
atlashome    315M  20.0T      3    412  4.19K  50.5M
atlashome   5.13G  20.0T      0  3.80K      0   477M
atlashome   8.84G  20.0T      0  3.73K      0   469M
atlashome   12.3G  20.0T      0  4.13K      0   518M
atlashome   15.6G  20.0T      0  4.16K      0   522M
atlashome   18.8G  20.0T      0  4.52K      0   567M
atlashome   18.8G  20.0T      0  3.82K      0   482M
atlashome   24.6G  20.0T      0  3.99K      0   497M
atlashome   24.6G  20.0T      0  4.41K      0   556M
atlashome   29.1G  20.0T      0  4.22K      0   529M
atlashome   33.6G  20.0T      0  3.88K      0   482M
atlashome   33.6G  20.0T      0  3.93K      0   495M
atlashome   37.7G  20.0T      0  3.88K      0   486M
atlashome   41.1G  20.0T      0  4.11K      0   516M
atlashome   44.7G  20.0T      0  4.03K      0   506M
atlashome   48.4G  20.0T      0  3.74K      0   470M
atlashome   51.3G  19.9T      0  4.38K      0   550M

Our current setup (raidz2)

  c0 c1 c4 c5 c6 c7
t7 3 3 3 3 3 3
t6 3 3 3 3 3 3
t5 2 2 3 3 3 3
t4 2 2 2 S 2 2
t3 2 2 2 2 2 2
t2 1 1 1 1 2 2
t1 1 1 1 1 1 1
t0 1 1 1 S 1 1

Command to create this: 
#!/bin/bash
ZPOOL=atlashome
zpool create -f $ZPOOL raidz2 c{0,1,4,6,7}t0d0 c{0,1,4,5,6,7}t1d0 c{0,1,4,5}t2d0  \
           raidz2 c{6,7}t2d0 c{0,1,4,5,6,7}t3d0 c{0,1,4,6,7}t4d0 c{0,1}t5d0 \
           raidz2 c{4,5,6,7}t5d0 c{0,1,5,4,6,7}t6d0 c{0,1,4,5,6,7}t7d0

This will create three raidz2 blocks striped across. This uses 40+6 disks for storage thus a net storage of 20 TB (500GB drives).

ALWAYS reserve space 

# create a small zfs file system with reservation, this will help to keep ZFS working once it's filled up
zfs create -o reservation=10M $ZPOOL/badtimes
-- CarstenAulbert - 12 Jun 2008
Edit  | Attach | Print version | History: r7 < r6 < r5 < r4 | Backlinks | View wiki text | Edit wiki text | More topic actions
Topic revision: r7 - 18 Jun 2008, HenningFehrmann
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback