Transfer files to and from Holodeck

There are multiple options to transfer files to and from Holodeck, each having their own advantages and disadvantages. If you are trying to get code onto Holodeck it is recommended to use git with a remote repository. For larger files and results a more direct way called gsisftp is recommended.

Git

Using git and a remote repository (like gitlab or github) has the great advantage, that your code is automatically version-controlled and you have a place to point others to, if they want to look at your code. Two disadvantages are the limited file size and the a large history log, when debugging the code. Therefore we recommend to use git when pushing finished code to the cluster and use gsisftp when pushing code for debugging purposes or retrieving results.

To use git with a remote repository, you need to create an account at gitlab or github (or any other remote git-repository service of your choice) and initiate the repository. (See here for a gitlab tutorial on how to setup a remote repository and here for a github tutorial)

After you have committed all your code to the remote repository log onto Holodeck and navigate to a directory, which should serve as the parent directory to your git-repository. There type

git clone <url_of_your_repository>

where you replace <url_of_your_repository> by the link you can get from your remote repository. This will download all the code to Holodeck. Now you can run it as you usually would.

For any following changes remember to commit and push them to the remote repository and use

git pull origin master

before executing your code in order to run the most up to date version of your code.

You can also make changes to your code on Holodeck and push those changes to the remote repository.

gsisftp

gsisftp is a tool to directly transfer files of any kind to and from Holodeck. sftp stands for safe file transfer protocol. To use it type

gsisftp holodeck1.aei.uni-hannover.de

to transfer files to and from holodeck1. You will see an interface that is a little different from your usual one. The header states “sftp>“ and does not show you, where on the system you are. In this environment you have only limited functionality. For instance “ls” does not take all flags. So in general you can try any command, but you should know that not all work.
What however does work are the commands “cd”, “ls”, “pwd”, “get” and “put”. The first three have actually two versions. If you prepend a small “L” to them (e.g. lcd) these commands act on your local system. It is noteworthy, that tab-complete and pattern matching with “*” work inside this environment.

This command in general, more or less, connects two directories directly. Usually it will connect the local directory, where your terminal was located in when executing the command, with your holohome folder. You can than only transfer files to and from your local directory to the current working directory on Holodeck.

Example:

Say I have a file called “test.txt” in my “/home/Documents” folder and want to transfer it to “/holohome/my.name/test”. I start my terminal at “~”, i.e. “/home” and type

gsisftp holodeck1.aei.uni-hannover.de

My local directory is now “/home” and the directory on Holodeck is “/holohome/my.name”. To find out where on each system I type

lpwd -> /home
pwd -> /holohome/my.name

I now need to change my local directory to be “/home/Documents”. To achieve this, I type

lcd Documents

Since I want to upload test.txt to “/holohome/my.name/test”, I need to change to the test directory on Holodeck. Hence I type

cd test

Everything is prepared now and I can upload the file by typing

put test.txt

For uploading use the “put” command, for downloading use the “get” command.

You can also up- or download entire directories by using the “-r” flag.

get -r <dir-name>
put -r <dir-name>

IMPORTANT

In case you experience authentication errors when downloading larger files (download stops after about 1-2GB and you get some error message telling you, that there is an authentication problem), try to connect to the node you are trying to copy files from using simple ssh (NOT gsissh) and when asked to continue type “yes”. You will than be asked to provide a password. At this point quit by pressing Ctrl + C and try downloading again.

-- MarlinSchFer - 03 Jun 2019