Attempt to collect input for all aspects of data migration from ZIB back to AEI Hannover

Prerequisites:

• Signed contract of cooperation between ZIB and AEI
  • Info Frau Roos 2011-10-14: we have paid ZIB in July for time span Jan/Jul 2011 (but no longer)
• Working Hierarchical Data Storage (HSM) system @Hannover - working
• Decision (by GEO team) which data to be transferred (and, if applicable, order of transfer)
  • Reply M.Hewitson 2011-10-14: everything

Past operation mode, ZIB:

• Data transfer via AEI Golm (morbo, using FHI link), later via AEI Hannover (charlie, using HLRN link) to trojanus
• Post-processing on trojanus, submit DMF requests to pompeius, staging to tapes (9940, 250GB) purchased by AEI until 2010
• as of June 2012, charlie cannot connect to ZIB anymore:

aeitrans@charlie:~$ traceroute titus.zib.de
traceroute to titus.zib.de (130.73.157.52), 30 hops max, 60 byte packets
 1  gate-c30-00-sn117.ruth.uni-hannover.de (130.75.117.250)  0.630 ms  0.625 ms  0.717 ms
 2  gate-s5-zib-vl1117.connect.uni-hannover.de (130.75.76.45)  1.834 ms  1.830 ms  1.932 ms
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
etc.
(In 2009, it looked like this:
aeitrans@charlie:~$ traceroute  trojanus.zib.de
traceroute to trojanus.zib.de (130.73.157.55), 30 hops max, 40 byte packets
 1  gate-c30-00-sn117.ruth.uni-hannover.de (130.75.117.250)  0.857 ms  0.964 ms  1.199 ms
 2  gate-s5-zib-vl1117.connect.uni-hannover.de (130.75.76.45)  2.067 ms  2.060 ms  2.182 ms
 3  * * *
 4  * * *
 5  trojanus.zib.de (130.73.157.55)  6.604 ms  6.599 ms  6.591 ms
"trojanus" has become "titus" since, but I don't have fresh traceroute output.)
   

Current status ZIB:

• Data transfer (until mid-May 2011) from AEI Hannover (charlie) to titus (virtual machine)
• Post-processing on titus, submit data into HSM "black box" with T10000 tapes owned by ZIB
• Data from "old" 9940 tapes migrated to T10000 by background process (transfer complete)
• Data transfer stopped in the middle of 2011/day

Who can be contacted at ZIB?

• (Manfred Stolle, our partner at ZIB, doesn't have access to the SAMFS server and the tape library!)
• Wolfgang Pyszkalski: head of IT services, would decide whether tapes may leave the archive
• Frau Steinke (administration) is in charge of contract issues (and may give orders to "technical" staff)
• Prof. Alexander Reinefeld may (?) override (or give orders to) administration
• For contact details, see ZIB staff page

Technical details:

• Data streams: raw data, h(t) reduced data set ("RDS"/"RDS3"), another reduced data set ("RDS9")
• Each stream consists of 1440 1-minute-long (frame) files, plus some trend files
• During post-processing, files are re-packed into hour archives (not to waste inodes on filesystem)

Data volume:

• "raw" frames: 144TB
• RDS3: 25TB
• RDS9: 87TB
• total: 260TB

HSM Hannover:

• Capacity
  • current:
    • 1800 LTO4 tapes (each 800GB uncompressed)
    • 256 450GB FC disks assembled into ~ 88TB of usable space (data-only and user data)
    • 280TB filesystem across 200 2TB drives (multiple 10 disk RAID6 stripes) (DDN - data direct network)
    • 960TB file systems for users' home directories (DDN - data direct network)
    • SAMFS license migrated to unlimited license
  • planned (next few months)
    • 300-400 extra LTO4 tapes - 600 procured + installed - current status updated
    • extra storage space to physically store tapes (+600 slots) - procured + installed - current status updated
    • then all ~ 2100 slots will be licensed - procured + installed - current status updated
    • 4 T10000C drives + 300 tapes (5TB each) - max out current drive configuration - procured + installed - current status updated
    • change meta data server to new license - procured + installed - current status updated
    • if money is available, more 2 or 3 TB drives for DDN system - procured + installed - current status updated

What does this mean, in terms of actions?

• Sign contract for 2011 with ZIB. Now. Tell them we're still interested in their expertise.
  • Even now that the renewed "contract" has already expired again, this is essential.
• Define which data sets are to be transferred to Hannover. (1st approx.: everything)
  • Everything. (M.Hewitson 2011-10-14)
• For tapes currently in use by Hannover HSM, get estimate of
  • number of tapes to hold all required data - if we stick with LTO4 and 2 copies: 650 tapes
  • total cost of those tapes (price max �30 per LTO4 tape, thus below �20k)
  • price for license extension to new capacity (new license will be governed by core of MDS server, not capacity anymore), total price for full system might be a large 5 digit number
• Transfer data to Hannover - either network or tapes
  • To ensure completeness and speed of transfer, variant B (below) is to be preferred!

Transfer strategies

1. Variant A: reverse archival data flow, pump everything through HLRN link
   • (very rough!) performance estimate: about 4 hours per day of data (all streams).
   • since data are not read from disk buffer throughput could be severely limited by tape mount times
   • we do not know anything about the arrangement of data on tapes!
   • Estimated time: 1 week per day, >5 years -> 9 months! (accurate to a factor of 3 or so)
   • Required manpower: internal: no estimate yet; external: none
     • We don't have any large disk space yet; if we get some 100GB per day of data, we can start some throughput measurements.
2. Variant B: move data physically
   • extract one copy of tapes from ZIB system, set read-only, import into Hannover system
     • buy tapes from ZIB, or only lend them?
     • what about 2nd copy if we buy the tapes from ZIB, what if there are tape errors?
   • requires transfer of metadata (tapes contain bit sequences without any description!)
   • there's a company with lots of expertise (three-letter abbreviation, can someone fill this in please? Perhaps HMK)
   • Carsten: cd /aeiperm; /opt/SUNWsamfs/sbin/samfsdump -q -f - | /usr/bin/gzip -c > dump.gz
     • Still has to be run by someone with root abilities on SAMfs server at ZIB!
   • perhaps rent/lease/purchase additional tape drive(s) matching ZIB's tape format
     • Is it correct that Hannover HSM has T10000 drives (same as ZIB)?
   • One run of repack/copy should create "own" data (in two copies). When done, export tapes and give back to ZIB
     • If we buy the tapes we don't have to repack, only to ensure they can be read properly
     • We might unpack everything to allow easier access to individual files (probably preferred by GEO people)
   • Estimated time: yet unknown
     • Actions to be performed in "foreground": import metadata, scan tape labels, - anything else?
     • Unpacking can be run in background (QA again!)
   • Required manpower: internal: almost none (is this true?); external: get offer

Well, first variant: Once set-up should be running smoothly, if staging attributes are set correctly, request to first file of a day will automatically pull in all other files of this day. Second variant might involve more manual work - even from our side. We already played with metadata dumps and restored files from older copies - might need discussion with ZIB if all GEO data is on its own FS.

Transfer speed tests:

• get raw/trends/2011/ (day001--day139), from tape:
  • elapsed: 43969 sec, transferred: 161184516 kB, rate 3665 kB/s
• get RDS3/daytrends/*/ (all years), from disk (1 file from tape already staged back):
  • elapsed: 107 sec, transferred: 2470516 kB, rate 23088 kB/s
• get raw/daytrends/*/:
  • elapsed: 5486 sec, transferred: 51497636 kB, rate 9387 kB/s

-- Steffen Grunewald - 14 Oct 2011