Running AMUSE with OpenMPI

AMUSE is developed with the MPICH2 package. It should also support OpenMPI but this is not tested as frequently. OpenMPI has some small differences and features to warrant this page.

Daemon

To run OpenMPI on a single node you do not have to start a daemon process like for MPICH2.

Open File Limits

When running all the test-cases a lot of processes are started and stopped. Current (1.3) versions of OpenMPI do not close all the pipes when a process is stopped and halfway through, the tests will fail with:

ORTE_ERROR_LOG: The system limit on number of pipes a
process can open was reached in file
../../../../../../orte/mca/odls/default/odls_default_module.c at line 218

A work-around for this problem is to change the limit for the maximum number of open files (ulimit -n or limit). This limit is often 1024 and that value is not high enough to run every unit test.

A tested and working value for the limit is 4096.

Unfortunately this is not very easy to do and very system dependent (you need to be root!)

For Ubuntu you can find a solution here:  http://ubuntuforums.org/showthread.php?t=638597

Openmpi without network connection

Openmpi will possibly not use the loopback device if no network is available. This results in processes not able to find each other (in principle shared memory transfers are available but apparently this does not work for processes spawned on the same machine). Solution is to set the following mca parameter: 'btl_tcp_if_exclude tcp' this can be set as an argument to mpirun: 'mpirun --mca btl_tcp_if_exclude tcp <amuse script command>' or in a file see e.g.  this explanation