Transfer files to and from Holodeck
There are multiple options to transfer files to and from Holodeck, each having their own advantages and disadvantages. If you are trying to get code onto Holodeck it is recommended to use git with a remote repository. For larger files and results a more direct way called gsisftp is recommended.
Using git and a remote repository (like gitlab
) has the great advantage, that your code is automatically version-controlled and you have a place to point others to, if they want to look at your code. Two disadvantages are the limited file size and the a large history log, when debugging the code. Therefore we recommend to use git when pushing finished code to the cluster and use gsisftp when pushing code for debugging purposes or retrieving results.
To use git with a remote repository, you need to create an account at gitlab
(or any other remote git-repository service of your choice) and initiate the repository. (See here
for a gitlab tutorial on how to setup a remote repository and here
for a github tutorial)
After you have committed all your code to the remote repository log onto Holodeck and navigate to a directory, which should serve as the parent directory to your git-repository. There type
git clone <url_of_your_repository>
where you replace <url_of_your_repository> by the link you can get from your remote repository. This will download all the code to Holodeck. Now you can run it as you usually would.
For any following changes remember to commit and push them to the remote repository and use
git pull origin master
before executing your code in order to run the most up to date version of your code.
You can also make changes to your code on Holodeck and push those changes to the remote repository.
gsisftp is a tool to directly transfer files of any kind to and from Holodeck. sftp stands for safe file transfer protocol. To use it type
to transfer files to and from holodeck1. You will see an interface that is a little different from your usual one. The header states “sftp>“ and does not show you, where on the system you are. In this environment you have only limited functionality. For instance “ls” does not take all flags. So in general you can try any command, but you should know that not all work.
What however does work are the commands “cd”, “ls”, “pwd”, “get” and “put”. The first three have actually two versions. If you prepend a small “L” to them (e.g. lcd) these commands act on your local system. It is noteworthy, that tab-complete and pattern matching with “*” work inside this environment.
This command in general, more or less, connects two directories directly. Usually it will connect the local directory, where your terminal was located in when executing the command, with your holohome folder. You can than only transfer files to and from your local directory to the current working directory on Holodeck.
Say I have a file called “test.txt” in my “/home/Documents” folder and want to transfer it to “/holohome/my.name/test”. I start my terminal at “~”, i.e. “/home” and type
My local directory is now “/home” and the directory on Holodeck is “/holohome/my.name”. To find out where on each system I type
lpwd -> /home
pwd -> /holohome/my.name
I now need to change my local directory to be “/home/Documents”. To achieve this, I type
Since I want to upload test.txt to “/holohome/my.name/test”, I need to change to the test directory on Holodeck. Hence I type
Everything is prepared now and I can upload the file by typing
For uploading use the “put” command, for downloading use the “get” command.
You can also up- or download entire directories by using the “-r” flag.
get -r <dir-name>
put -r <dir-name>
In case you experience authentication errors when downloading larger files (download stops after about 1-2GB and you get some error message telling you, that there is an authentication problem), try to connect to the node you are trying to copy files from using simple ssh (NOT gsissh) and when asked to continue type “yes”. You will than be asked to provide a password. At this point quit by pressing Ctrl + C and try downloading again.
- 03 Jun 2019