Running amuse on the CINECA super computer

This super computer consists of AIX 6.1.0.0, powerpc 6 nodes with xlc/xlf compilers.

At the time of writing this description not all codes could be build on AIX, most codes need different flags and some other settings to get it to work.

Preparation

Make a directory for all work files etc.

mkdir amuse

cd amuse

Get the latest amuse release

wget http://www.amusecode.org/releases/amuse-5.0.tar.gz

To get an svn version of AMUSE (if you have an svn account):

module load subversion

svn co http://www.amusecode.org/svn/trunk amuse-svn

For a release download, unzip and untar the release

gzip -d amuse-5.0.tar.gz

tar -xvf amuse-5.0.tar

Create a file called 'setdev' and put this text in it:

#!/bin/sh

module load python
module load numpy
module load szlib
module load zlib
module load hdf5
module load fftw/3.2.2--xl--10.1
module load gsl
module load cmake
module load hdf5
module load cfitsio
module load make

export AMUSEROOT=/sp6/userhpe/<username>/<directory>

export PYTHONPATH=${PYTHONPATH}:${AMUSEROOT}/amuse-5.0/src
export PYTHONPATH=${AMUSEROOT}/prerequisites/lib/python2.6/site-packages/:${PYTHONPATH}

export PREFIX=${AMUSEROOT}/prerequisites/
export PATH=${PREFIX}/bin:${PATH}
export LD_LIBRARY_PATH=${PREFIX}/lib:${LD_LIBRARY_PATH}
export LIBRARY_PATH=${PREFIX}/lib:${LIBRARY_PATH}

export FC=xlf90
export F77=xlf
export FORTRAN=xlf90
export CC=xlc
export CXX=xlc++

Take care to replace <username> with your name and <directory> with the directory containing the workfiles ('amuse' if you follow all steps to the letter). Also take car to replace amuse-5.0 with amuse-svn if you ran the svn checkout line.

work

source setdev

mkdir $PREFIX

./configure CC=gxlc FC=gxlf90 CXX=gxlc++ MPICC="mpcc -qlanglvl=stdc99" MPICXX="mpCC -cpp"

Nosetests and docutils

Nose and docutils are straightforward python install packages. Download and untar the packages ( Nose,  docutils ) and type

python setup.py build
python setup.py install --prefix=$PREFIX

from their respective directories.

building hdf5

source setdev

mkdir $PREFIX

cd $PREFIX

mkdir download

wget http://h5py.googlecode.com/files/h5py-2.0.1.tar.gz

gzip -d h5py-2.0.1.tar

tar -xvf h5py-2.0.1.tar

cd h5py-2.0.1

To build h5py you need to change some setting in the setup.py file

In line 60 change 'libraries' : ['hdf5'], to 'libraries' : ['hdf5','z','sz'],

In line 62 change the library dirs from 'library_dirs' : [] to 'library_dirs' : ['/cineca/prod/libraries/szlib/2.1/xl--10.1/lib'],

export HDF5_DIR=$HDF5_HOME; python ./configure.py

python setup.py build

run again

python setup.py build

it will complain with::

ld: 0711-418 ERROR: Import or export file h5g.exp at line 11:
        A symbol name may only be followed by an export attribute
        or an address. The line is being ignored.

now for some difficult steps

cp /cineca/prod/tools/python/2.6.2/xl--10.1/lib/python2.6/config/ld_so_aix .
cp /cineca/prod/tools/python/2.6.2/xl--10.1/lib/python2.6/config/makexp_aix . 

edit the ld_so_aix file and put a comment before "rm -f $expfile" so #rm -f $expfile

make build setup.py

will complain again, but now copy the link line and change /cineca/prod/tools/python/2.6.2/xl--10.1/lib/python2.6/config/ld_so_aix to ./ld_so_aix

link by hand:

/ld_so_aix xlc_r -bI:/cineca/prod/tools/python/2.6.2/xl--10.1/lib/python2.6/config/python.exp build/temp.aix-6.1-2.6/h5py/h5g.o -L/cineca/prod/libraries/szlib/2.1/xl--10.1/lib -L/cineca/prod/libraries/hdf5/1.8.4_ser/xl--10.1/lib -R/cineca/prod/libraries/szlib/2.1/xl--10.1/lib -R/cineca/prod/libraries/hdf5/1.8.4_ser/xl--10.1/lib -lhdf5 -lz -lsz -o build/lib.aix-6.1-2.6/h5py/h5g.so

Still an error but now you have the h5g.exp file

Open it and remove line 11 (with the <long double in it>

edit ld_so_aix and comment the line where makexp is called (line 179) #$makexp $expfile "$objfile" $objs

and call ld_so_aix again:

/ld_so_aix xlc_r -bI:/cineca/prod/tools/python/2.6.2/xl--10.1/lib/python2.6/config/python.exp build/temp.aix-6.1-2.6/h5py/h5g.o -L/cineca/prod/libraries/szlib/2.1/xl--10.1/lib -L/cineca/prod/libraries/hdf5/1.8.4_ser/xl--10.1/lib -R/cineca/prod/libraries/szlib/2.1/xl--10.1/lib -R/cineca/prod/libraries/hdf5/1.8.4_ser/xl--10.1/lib -lhdf5 -lz -lsz -o build/lib.aix-6.1-2.6/h5py/h5g.so

now it should finish without complaining and you can do:

python setup.py build
python setup.py install --prefix=$PREFIX

}}}