Attempt to collect input for all aspects of data migration from ZIB back to AEI Hannover


  • Signed contract of cooperation between ZIB and AEI
    • Info Frau Roos 2011-10-14: we have paid ZIB in July for time span Jan/Jul 2011 (but no longer)
  • Working Hierarchical Data Storage (HSM) system @Hannover - working
  • Decision (by GEO team) which data to be transferred (and, if applicable, order of transfer)
    • Reply M.Hewitson 2011-10-14: everything

Past operation mode, ZIB:

  • Data transfer via AEI Golm (morbo, using FHI link), later via AEI Hannover (charlie, using HLRN link) to trojanus
  • Post-processing on trojanus, submit DMF requests to pompeius, staging to tapes (9940, 250GB) purchased by AEI until 2010
  • as of June 2012, charlie cannot connect to ZIB anymore:
aeitrans@charlie:~$ traceroute
traceroute to (, 30 hops max, 60 byte packets
 1 (  0.630 ms  0.625 ms  0.717 ms
 2 (  1.834 ms  1.830 ms  1.932 ms
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
(In 2009, it looked like this:
aeitrans@charlie:~$ traceroute
traceroute to (, 30 hops max, 40 byte packets
 1 (  0.857 ms  0.964 ms  1.199 ms
 2 (  2.067 ms  2.060 ms  2.182 ms
 3  * * *
 4  * * *
 5 (  6.604 ms  6.599 ms  6.591 ms
"trojanus" has become "titus" since, but I don't have fresh traceroute output.)

Current status ZIB:

  • Data transfer (until mid-May 2011) from AEI Hannover (charlie) to titus (virtual machine)
  • Post-processing on titus, submit data into HSM "black box" with T10000 tapes owned by ZIB
  • Data from "old" 9940 tapes migrated to T10000 by background process (transfer complete)
  • Data transfer stopped in the middle of 2011/day

Who can be contacted at ZIB?

  • (Manfred Stolle, our partner at ZIB, doesn't have access to the SAMFS server and the tape library!)
  • Wolfgang Pyszkalski: head of IT services, would decide whether tapes may leave the archive
  • Frau Steinke (administration) is in charge of contract issues (and may give orders to "technical" staff)
  • Prof. Alexander Reinefeld may (?) override (or give orders to) administration
  • For contact details, see ZIB staff page

Technical details:

  • Data streams: raw data, h(t) reduced data set ("RDS"/"RDS3"), another reduced data set ("RDS9")
  • Each stream consists of 1440 1-minute-long (frame) files, plus some trend files
  • During post-processing, files are re-packed into hour archives (not to waste inodes on filesystem)

Data volume:

  • "raw" frames: 144TB
  • RDS3: 25TB
  • RDS9: 87TB
  • total: 260TB

HSM Hannover:

  • Capacity
    • current:
      • 1800 LTO4 tapes (each 800GB uncompressed)
      • 256 450GB FC disks assembled into ~ 88TB of usable space (data-only and user data)
      • 280TB filesystem across 200 2TB drives (multiple 10 disk RAID6 stripes) (DDN - data direct network)
      • 960TB file systems for users' home directories (DDN - data direct network)
      • SAMFS license migrated to unlimited license
    • planned (next few months)
      • 300-400 extra LTO4 tapes - 600 procured + installed - current status updated
      • extra storage space to physically store tapes (+600 slots) - procured + installed - current status updated
      • then all ~ 2100 slots will be licensed - procured + installed - current status updated
      • 4 T10000C drives + 300 tapes (5TB each) - max out current drive configuration - procured + installed - current status updated
      • change meta data server to new license - procured + installed - current status updated
      • if money is available, more 2 or 3 TB drives for DDN system - procured + installed - current status updated

What does this mean, in terms of actions?

  • Sign contract for 2011 with ZIB. Now. Tell them we're still interested in their expertise.
    • Even now that the renewed "contract" has already expired again, this is essential.
  • Define which data sets are to be transferred to Hannover. (1st approx.: everything)
    • Everything. (M.Hewitson 2011-10-14)
  • For tapes currently in use by Hannover HSM, get estimate of
    • number of tapes to hold all required data - if we stick with LTO4 and 2 copies: 650 tapes
    • total cost of those tapes (price max �30 per LTO4 tape, thus below �20k)
    • price for license extension to new capacity (new license will be governed by core of MDS server, not capacity anymore), total price for full system might be a large 5 digit number
  • Transfer data to Hannover - either network or tapes
    • To ensure completeness and speed of transfer, variant B (below) is to be preferred!

Transfer strategies

  1. Variant A: reverse archival data flow, pump everything through HLRN link
    • (very rough!) performance estimate: about 4 hours per day of data (all streams).
    • since data are not read from disk buffer throughput could be severely limited by tape mount times
    • we do not know anything about the arrangement of data on tapes!
    • Estimated time: 1 week per day, >5 years -> 9 months! (accurate to a factor of 3 or so)
    • Required manpower: internal: no estimate yet; external: none
      • We don't have any large disk space yet; if we get some 100GB per day of data, we can start some throughput measurements.
  2. Variant B: move data physically
    • extract one copy of tapes from ZIB system, set read-only, import into Hannover system
      • buy tapes from ZIB, or only lend them?
      • what about 2nd copy if we buy the tapes from ZIB, what if there are tape errors?
    • requires transfer of metadata (tapes contain bit sequences without any description!)
    • there's a company with lots of expertise (three-letter abbreviation, can someone fill this in please? Perhaps HMK)
    • Carsten: cd /aeiperm; /opt/SUNWsamfs/sbin/samfsdump -q -f - | /usr/bin/gzip -c > dump.gz
      • Still has to be run by someone with root abilities on SAMfs server at ZIB!
    • perhaps rent/lease/purchase additional tape drive(s) matching ZIB's tape format
      • Is it correct that Hannover HSM has T10000 drives (same as ZIB)?
    • One run of repack/copy should create "own" data (in two copies). When done, export tapes and give back to ZIB
      • If we buy the tapes we don't have to repack, only to ensure they can be read properly
      • We might unpack everything to allow easier access to individual files (probably preferred by GEO people)
    • Estimated time: yet unknown
      • Actions to be performed in "foreground": import metadata, scan tape labels, - anything else?
      • Unpacking can be run in background (QA again!)
    • Required manpower: internal: almost none (is this true?); external: get offer

Well, first variant: Once set-up should be running smoothly, if staging attributes are set correctly, request to first file of a day will automatically pull in all other files of this day. Second variant might involve more manual work - even from our side. We already played with metadata dumps and restored files from older copies - might need discussion with ZIB if all GEO data is on its own FS.

Transfer speed tests:

  • get raw/trends/2011/ (day001--day139), from tape:
    • elapsed: 43969 sec, transferred: 161184516 kB, rate 3665 kB/s
  • get RDS3/daytrends/*/ (all years), from disk (1 file from tape already staged back):
    • elapsed: 107 sec, transferred: 2470516 kB, rate 23088 kB/s
  • get raw/daytrends/*/:
    • elapsed: 5486 sec, transferred: 51497636 kB, rate 9387 kB/s

-- Steffen Grunewald - 14 Oct 2011
Topic revision: r8 - 27 Jun 2012, CarstenAulbert
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback