Running AMUSE on The Cartesius supercomputer

The Cartesius is the Dutch national supercomputer. For more information on e.g. obtaining an account, see the SURFSara website: https://www.surfsara.nl/systems/cartesius.

Using AMUSE on the Cartesius is relatively straightforward. Below is a tested method for installing and using AMUSE using the prerequisites, though other options (e.g. using only software pre-installed on the machine) should also be possible. For a generic description of the installation of prerequisites, see the Installation of the prerequisite software section.

Obtaining AMUSE

We assume a copy of AMUSE has been downloaded to an amuse folder in the users home. For instance using svn:

> cd /home/USERNAME

> svn co http://www.amusecode.org/svn/trunk amuse

Environment settings

Since we will be using the AMUSE prerequisites software, we need to set some environment variables. Make sure the following lines are present in your .bashrc file:

#file: ~/.bashrc

#load java 1.7 needed for some codes
module load java/oracle

export PREFIX=~/amuse/prerequisites
export PATH=${PREFIX}/bin:${PATH}
export LD_LIBRARY_PATH=${PREFIX}/lib:${LD_LIBRARY_PATH}

#use gfortran
export FC=gfortran
export F77=gfortran

Also make sure that .bashrc is loaded in your .bash_profile. This way, the enviroment is always set correctly, both in interactive and non-interactive mode.

#file: ~/.bash_profile

if [ -f ~/.bashrc ]; then
    . ~/.bashrc
fi

Note: be sure to re-connect to the machine for these changes to take effect.

Install AMUSE prerequisites

Next, we will install all prerequisites of amuse using the AMUSE supplied scripts.

# create a directory for the prerequisites
> mkdir ~/amuse/prerequisites

# go to the <doc/install> directory
> cd ~/amuse/doc/install

# Start the installation script for Python.
> ./install-python.sh

# Download the prerequisite packages.
> ./install.py download

# Install prerequisites. Use hydra as the default MPI process manager. May take a while...
> ./install.py --hydra install

# Optionally also install matplotlib
> ./install.py --matplotlib install

Configure and build AMUSE

Configuring and building amuse is now as normal.

# go to the amuse directory
> cd ~/amuse

# configure amuse
> ./configure MPIEXEC=mpiexec.hydra

# build amuse
> make

# optionally also install codes requiring downloading files
> make DOWNLOAD_CODES=1

Test the installation

To test your AMUSE installation, run nosetests.

Note: do not run simulations on the frontend of the cartesius. This is not allowed!

# go to the amuse directory
> cd ~/amuse

> mpiexec.hydra -n 1 nosetests -v tests

Running on a Cartesius node

Running on the Cartesius is typically done by submitting a slurm script. See the surfsare site for more info:

https://www.surfsara.nl/systems/cartesius/usage/batch-usage

Below is a simple example script for running amuse on Cartesius.

#!/bin/bash
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -p short
#SBATCH -t 10

cd ~/amuse

mpiexec.hydra -n 1 ./amuse.sh examples/syllabus/gravity_simple.py

Submit using sbatch, get status using squeue, cancel using scancel

#submit a script
> sbatch example-script

#list jobs of current user
> squeue -u $USER

#cancel job 505224
scancel 505224

#cancel all jobs of the current user
> scancel -u $USER

Using multiple nodes should also work, by specifying this to slurm. MPI will automatically pickup on this and spread workers over all nodes.

#!/bin/bash
#SBATCH -N 2
#SBATCH -n 10
#SBATCH -p short
#SBATCH -t 10

cd ~/amuse

#note that this simple example uses only a single worker
mpiexec.hydra -n 1 ./amuse.sh examples/syllabus/gravity_simple.py