Chances are you need to run your HPC job against a dataset, perhaps quite a sizable one. There are a number of places to store data on Spartan while you're working with it, and ways to get data in and out.
Not for Long-Term Storage
While it's often essential to have fast nearby storage while working on your data, please don't use Spartan as your long-term data repository. It's not designed for that, may not conform to the requirements set by your institution or funding body, and we don't guarantee to store your data indefinitely (though we certainly won't get rid of it without asking you first).
Data and Storage Solutions Beyond Spartan
The University offers a range of other data storage and management solutions to meet your needs, beyond the short-term storage available on Spartan, which are described here.
In some cases it's possible to integrate these resources with your account on Spartan to streamline your workflow. Get in touch if you'd like to find out more for your particular application.
Transferring Files to Spartan
There is a mismatched MTU between the Spartan physical nodes and their VLAN, which will be fixed in the next maintenance window. If downloads to Spartan are not working it is recommended that a proxy is established before using
curl etc. For example:
wget --no-check-certificate ftp://ftp.broadinstitute.org/distribution/83RC2_q_image.tgz
Where to Store Your Data on Spartan
Your projects directory is the best place to store research data while you're working on it. It's located at
Others in your project can access it, and 500 GB of storage is available per project. If you need more than this, get in touch and we'll try to find a solution.
Your home directory, i.e.
/home/yourusername can be used to store small amounts of data, however this is generally discouraged. It's best suited to short-lived and non-critical data, for example while working through our getting started tutorial or testing out new software.
Others in your project won't have access, and you're limited to 50GB of storage.
You can store temporary working data while your job is running at
/scratch/. This is handy if your job generates large files while it's running that you don't need to keep. Total scratch space is limited to 8TB, shared among all users.
How to Transfer Data In and Out of Spartan
Secure Copy (scp)
You can use the
scp command to move data from your local machine to Spartan. For example, to move
mydata.dat from your current working directory to your project directory on Spartan:
$ scp local.dat firstname.lastname@example.org:/data/projects/myproject/remote.dat
You can transfer files from Spartan to your local machine by reversing the order of the arguments like so:
$ scp email@example.com:/data/projects/myproject/remote.dat local.dat
For Windows users, PuTTY provides an equivalent tool called
pscp. If you're data is located on a remote machine, you can SSH into that system first, and then use
scp from that machine to transfer your data into Spartan.
Repeatedly transferring large files in and out of Spartan via
scp can be tedious. A good alternative is rsync, which only transfers the parts that have changed. It can work on single files, or whole directories, and the syntax is much same as for
$ rsync local.dat firstname.lastname@example.org:/data/projects/myproject/remote.dat
Note that the first argument is the source, and the second is the destination which will be modified to match the source.