Attempt to collect input for all aspects of data migration from ZIB back to AEI Hannover
- Signed contract of cooperation between ZIB and AEI
- Info Frau Roos 2011-10-14: we have paid ZIB in July for time span Jan/Jul 2011 (but no longer)
- Working Hierarchical Data Storage (HSM) system @Hannover - working
- Decision (by GEO team) which data to be transferred (and, if applicable, order of transfer)
- Reply M.Hewitson 2011-10-14: everything
Past operation mode, ZIB:
- Data transfer via AEI Golm (morbo, using FHI link), later via AEI Hannover (charlie, using HLRN link) to trojanus
- Post-processing on trojanus, submit DMF requests to pompeius, staging to tapes (9940, 250GB) purchased by AEI until 2010
- as of June 2012, charlie cannot connect to ZIB anymore:
aeitrans@charlie:~$ traceroute titus.zib.de
traceroute to titus.zib.de (18.104.22.168), 30 hops max, 60 byte packets
1 gate-c30-00-sn117.ruth.uni-hannover.de (22.214.171.124) 0.630 ms 0.625 ms 0.717 ms
2 gate-s5-zib-vl1117.connect.uni-hannover.de (126.96.36.199) 1.834 ms 1.830 ms 1.932 ms
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
(In 2009, it looked like this:
aeitrans@charlie:~$ traceroute trojanus.zib.de
traceroute to trojanus.zib.de (188.8.131.52), 30 hops max, 40 byte packets
1 gate-c30-00-sn117.ruth.uni-hannover.de (184.108.40.206) 0.857 ms 0.964 ms 1.199 ms
2 gate-s5-zib-vl1117.connect.uni-hannover.de (220.127.116.11) 2.067 ms 2.060 ms 2.182 ms
3 * * *
4 * * *
5 trojanus.zib.de (18.104.22.168) 6.604 ms 6.599 ms 6.591 ms
"trojanus" has become "titus" since, but I don't have fresh traceroute output.)
Current status ZIB:
- Data transfer (until mid-May 2011) from AEI Hannover (charlie) to titus (virtual machine)
- Post-processing on titus, submit data into HSM "black box" with T10000 tapes owned by ZIB
- Data from "old" 9940 tapes migrated to T10000 by background process (transfer complete)
- Data transfer stopped in the middle of 2011/day
- (Manfred Stolle, our partner at ZIB, doesn't have access to the SAMFS server and the tape library!)
- Wolfgang Pyszkalski: head of IT services, would decide whether tapes may leave the archive
- Frau Steinke (administration) is in charge of contract issues (and may give orders to "technical" staff)
- Prof. Alexander Reinefeld may (?) override (or give orders to) administration
- For contact details, see ZIB staff page
- Data streams: raw data, h(t) reduced data set ("RDS"/"RDS3"), another reduced data set ("RDS9")
- Each stream consists of 1440 1-minute-long (frame) files, plus some trend files
- During post-processing, files are re-packed into hour archives (not to waste inodes on filesystem)
- "raw" frames: 144TB
- RDS3: 25TB
- RDS9: 87TB
- total: 260TB
- 1800 LTO4 tapes (each 800GB uncompressed)
- 256 450GB FC disks assembled into ~ 88TB of usable space (data-only and user data)
- 280TB filesystem across 200 2TB drives (multiple 10 disk RAID6 stripes) (DDN - data direct network)
- 960TB file systems for users' home directories (DDN - data direct network)
- SAMFS license migrated to unlimited license
- planned (next few months)
- 300-400 extra LTO4 tapes - 600 procured + installed - current status updated
- extra storage space to physically store tapes (+600 slots) - procured + installed - current status updated
- then all ~ 2100 slots will be licensed - procured + installed - current status updated
- 4 T10000C drives + 300 tapes (5TB each) - max out current drive configuration - procured + installed - current status updated
- change meta data server to new license - procured + installed - current status updated
- if money is available, more 2 or 3 TB drives for DDN system - procured + installed - current status updated
What does this mean, in terms of actions?
- Sign contract for 2011 with ZIB. Now. Tell them we're still interested in their expertise.
- Even now that the renewed "contract" has already expired again, this is essential.
- Define which data sets are to be transferred to Hannover. (1st approx.: everything)
- Everything. (M.Hewitson 2011-10-14)
- For tapes currently in use by Hannover HSM, get estimate of
- number of tapes to hold all required data - if we stick with LTO4 and 2 copies: 650 tapes
- total cost of those tapes (price max �30 per LTO4 tape, thus below �20k)
- price for license extension to new capacity (new license will be governed by core of MDS server, not capacity anymore), total price for full system might be a large 5 digit number
- Transfer data to Hannover - either network or tapes
- To ensure completeness and speed of transfer, variant B (below) is to be preferred!
- Variant A: reverse archival data flow, pump everything through HLRN link
- (very rough!) performance estimate: about 4 hours per day of data (all streams).
- since data are not read from disk buffer throughput could be severely limited by tape mount times
- we do not know anything about the arrangement of data on tapes!
- Estimated time: 1 week per day, >5 years -> 9 months! (accurate to a factor of 3 or so)
- Required manpower: internal: no estimate yet; external: none
- We don't have any large disk space yet; if we get some 100GB per day of data, we can start some throughput measurements.
- Variant B: move data physically
- extract one copy of tapes from ZIB system, set read-only, import into Hannover system
- buy tapes from ZIB, or only lend them?
- what about 2nd copy if we buy the tapes from ZIB, what if there are tape errors?
- requires transfer of metadata (tapes contain bit sequences without any description!)
- there's a company with lots of expertise (three-letter abbreviation, can someone fill this in please? Perhaps HMK)
cd /aeiperm; /opt/SUNWsamfs/sbin/samfsdump -q -f - | /usr/bin/gzip -c > dump.gz
- Still has to be run by someone with root abilities on SAMfs server at ZIB!
- perhaps rent/lease/purchase additional tape drive(s) matching ZIB's tape format
- Is it correct that Hannover HSM has T10000 drives (same as ZIB)?
- One run of repack/copy should create "own" data (in two copies). When done, export tapes and give back to ZIB
- If we buy the tapes we don't have to repack, only to ensure they can be read properly
- We might unpack everything to allow easier access to individual files (probably preferred by GEO people)
- Estimated time: yet unknown
- Actions to be performed in "foreground": import metadata, scan tape labels, - anything else?
- Unpacking can be run in background (QA again!)
- Required manpower: internal: almost none (is this true?); external: get offer
Well, first variant: Once set-up should be running smoothly, if staging attributes are set correctly, request to first file of a day will automatically pull in all other files of this day. Second variant might involve more manual work - even from our side. We already played with metadata dumps and restored files from older copies - might need discussion with ZIB if all GEO data is on its own FS.
Transfer speed tests:
- get raw/trends/2011/ (day001--day139), from tape:
elapsed: 43969 sec, transferred: 161184516 kB, rate 3665 kB/s
- get RDS3/daytrends/*/ (all years), from disk (1 file from tape already staged back):
elapsed: 107 sec, transferred: 2470516 kB, rate 23088 kB/s
- get raw/daytrends/*/:
elapsed: 5486 sec, transferred: 51497636 kB, rate 9387 kB/s
-- Steffen Grunewald - 14 Oct 2011