Einstein@home RAID setup testing
Over time the einstein-abp1 download server (mostly BRP project) went through various updates to increase download throughput (apache2-mpm-worker, lighttpd, nginx on the web server side, move from 8bit to 4bit sampled data, deadline scheduler optimiztion) - however we seem to hit a limit around 900k downloads a day with pre-procesing writing new files to the machine at the same speed as it's deleted, up to 1.6TB/day (2050kB files + housekeeping files). Next logical step before going to a clustered FS or similar solution is to go with more download servers, each running its own work unit generator. Going that route we also looked briefly into an alternative to the current 16 disk Areca 1261ML RAID6 setup.
Web server tests
The machine was re-installed and a test data set created (1024 download directories each with 2048 2050kB files plus an md5 file for each data file). Webserver was nginx with varying number of workers, each worker accepting up to 512 connections. Underneath was either a RAID6 or RADI10 setup with 16 750 GB Hitachi Ultradrives. The test server was connected by a single 1
GbE connection while the up to 9 clients were connected via 10
GbE to the network. Clients would randomly download any of the created files along with the md5 file for comparison (done on the client), in total 2502 data and 2502 md5 files were downloaded each run (~5GB volume). web server host was limited to 1000MB memory.
raidlevel: |
6 |
10 |
deadline: |
default |
optimized |
default |
optimized |
nginx worker: |
1 |
1 |
2 |
4 |
8 |
1 |
1 |
4 |
8 |
num clients |
(run time in s, 5009 MB downloaded each time) |
1 |
434 |
225 |
237 |
225 |
226 |
382 |
246 |
252 |
244 |
2 |
380 |
180 |
177 |
166 |
153 |
346 |
204 |
174 |
159 |
3 |
388 |
183 |
164 |
136 |
122 |
334 |
207 |
141 |
132 |
4 |
375 |
174 |
133 |
115 |
106 |
323 |
198 |
119 |
106 |
5 |
372 |
178 |
117 |
106 |
97 |
154 |
190 |
108 |
97 |
6 |
375 |
160 |
110 |
99 |
89 |
395 |
189 |
105 |
89 |
7 |
373 |
158 |
105 |
92 |
91 |
310 |
177 |
93 |
85 |
8 |
378 |
149 |
102 |
102 |
82 |
315 |
171 |
87 |
79 |
9 |
375 |
158 |
99 |
84 |
82 |
274 |
89 |
83 |
76 |
labels |
r6-def-1 |
r6-opt-1 |
r6-opt-2 |
r6-opt-4 |
r6-opt-8 |
r10-def-1 |
r10-opt-1 |
r10-opt-4 |
r10-opt-8 |
Write tests
Here we created a data set with 32 directories and 256 files each. Each file again 2050kB large along with a small md5 file.
conclusion
RAID10 seems to be marginally better in these synthetic tests, as we have extensive experience with such a server under RAID6, we will opt for RAID10 for next one.
--
CarstenAulbert - 11 Sep 2012