Supported File Formats

File input and output

AMUSE can read and write to/from several data formats. The main routines are write_set_to_file and read_set_from_file, explained below.

Introduction

The AMUSE framework provides a generic way of reading and writing sets of entities (these entities can be particles, gasclouds or gridpoints). AMUSE provides a function to write a set to a file and a function to read a set from a file. These functions need the name of the file and the format of the data in the file. We will describe these functions first in this chapter. The functions can throw 3 kinds of exceptions, these are described next.

Use

To write a data set to a space separated text file do:

>>> from amuse.io import write_set_to_file
>>> from amuse.datamodel.core import Particles
>>> from amuse.units import units
>>> x = Particles(100)
>>> x.mass = 10.0 | units.MSun
>>> x.radius = range(1.0,101.0) | units.RSun
>>> write_set_to_file(x, "test.csv","txt", attribute_types = [units.MSun, units.RSun])
amuse.io.write_set_to_file(set, filename, format='csv', **format_specific_keyword_arguments)

Write a set to the given file in the given format.

Parameters:
  • filename – name of the file to write the data to
  • format – name of a registered format or a FileFormatProcessor subclass (must be a class and not an instance)

All other keywords are set as attributes on the fileformat processor. To determine the supported options for a processor call get_options_for_format()

Registered file formats:

amuse,
Process an HDF5 file
csv,
Process comma separated files
dyn,
Process a Starlab binary structured file
gadget,
Process a Gadget binary data file
hdf5,
Process an HDF5 file
nemo,
Process a NEMO binary structured file
starlab,
Process a Starlab binary structured file
tsf,
Process a NEMO binary structured file
txt,
Process a text file containing a table of values separated by a predefined character
vts,
Process a text file containing a table of values separated by a predefined character
vtu,
Process a text file containing a table of values separated by a predefined character
amuse.io.read_set_from_file(filename, format='csv', **format_specific_keyword_arguments)

Read a set from the given file in the given format.

Parameters:
  • filename – name of the file to read the data from
  • format – name of a registered format or a FileFormatProcessor subclass (must be a class and not an instance)

All other keywords are set as attributes on the fileformat processor. To determine the supported options for a processor call get_options_for_format()

Registered file formats:

amuse,
Process an HDF5 file
csv,
Process comma separated files
dyn,
Process a Starlab binary structured file
gadget,
Process a Gadget binary data file
hdf5,
Process an HDF5 file
nemo,
Process a NEMO binary structured file
starlab,
Process a Starlab binary structured file
tsf,
Process a NEMO binary structured file
txt,
Process a text file containing a table of values separated by a predefined character
vts,
Process a text file containing a table of values separated by a predefined character
vtu,
Process a text file containing a table of values separated by a predefined character
amuse.io.get_options_for_format(format='csv')

Retuns a list of tuples, each tuple contains the name of the option, a description of the option and the default values.

Parameters:format – name of a registered format or a FileFormatProcessor subclass (must be a class and not an instance)

Exceptions

class amuse.io.UnsupportedFormatException(*arguments)

Raised when the given format is not supported by AMUSE.

class amuse.io.CannotSaveException(*arguments)

Raised when the given format cannot save data (only reading of data is supported for the format)

class amuse.io.CannotLoadException(*arguments)

Raised when the given format cannot read data (only saving of data is supported for the format)

Amuse HDF5 files

AMUSE provides it’s own data format based on hdf5 files. The HDF5 file storage provides support for storing particle set and grids in the same file. It also retains the particle references and can store multiple versions of the same set.

>>> from amuse.support import io
>>> from amuse.ic.plummer import new_plummer_model
>>> particles = new_plummer_model(1000)
>>> io.write_set_to_file(
            particles,
           'plummer.hdf5',
           'hdf5',
    )
>>> io.read_set_from_file('plummer.hdf5', 'hdf5')
extra_attributes
Extra attributes to store with the data set. Some formats (moste notably the amuse native format) can store extra attributes with the set in file. The ‘write_set_to_file’ function will collect all keyword arguments that do not match to an option into the extra attributes dictionary.
append_to_file
If set to True, new data is appended to HDF5 files. If set to False, the existing file is removed and overwritten. Only relevant for write set to file. (default: True)
copy_history
If set to True, the savepoint history is read from file into memory. By default the history will be kept on file and the file will be kept open
close_file
If set to True, the file is closed after reading, unless you set copy_history to True no previous versions will be returned
return_working_copy
If set to True, return a working copy in memory you can manipulate, savepoint etc. Only available for version 2.0 in version 1.0 a working copy (i.e. a particles set or grid in memory) is always returned. (default: False)
version
AMUSE storage version to use, needs to be > ‘2.0’ if you want to store links between particles and grids (default: ‘1.0’)
names
A list of names to save the data under. If filled this will load the sets or grids with the given names and return this as a list. When saving the names will be used to save each set, a list of sets is needed in the write_set_to_file function (default: [])
return_context
If set to True, will return a context manager instead of the loaded set(s). This context manager will take care of properly closing any connected resources. To access the set, use the with statement. .. code-block:: python with load_set_from_file(“example.h5”, “amuse”) as particles: print particles Usefull for cases where close_file == False. (default: False)

Text files

AMUSE has support for reading and writing text (.txt, .csv) files. You can specify the txt format by entering “txt” (for space separated files) or “csv” (for comma separated files) as the format:

>>> from amuse.support import io
>>> particles = io.read_set_from_file(
           'plummer.txt',
           'txt',
           attribute_types = (units.m, units.m, units.kms),
           attribute_names= ('x', 'y', 'vx')
    )
>>> io.write_set_to_file(particles, 'plummer.txt', 'txt')
attribute_names
List of the names of the attributes to load or store
key_in_column
Column for the key (default is -1, no key stored/loaded). Keys will be interleaved with the data if set other than 0. No attribute type or name can be given for the key.
attribute_types
List of the units of the attributes to store. If not given the units will be derived from the units stored in the attribute set. When derived from the set, the units will always be converted to the base units.
extra_attributes
Extra attributes to store with the data set. Some formats (moste notably the amuse native format) can store extra attributes with the set in file. The ‘write_set_to_file’ function will collect all keyword arguments that do not match to an option into the extra attributes dictionary.
column_separator
Separator between the columns
attribute_dtypes
List of the data types of the attributes. If not given, the data types will be float64.
precision_of_number_output
The precision is a decimal number indicating how many digits should be displayed after the decimal point
header_prefix_string
Lines starting with this character will be handled as part of the header
footer_prefix_string
Lines starting with this character will be handled as part of the footer
must_store_units_in_header
Store line with parsable units in the header of the text file. This line can be used to restore the units on reading the file. When this line is added to the file you do not need to specify the attribute_types on reading the file
maximum_number_of_lines_buffered
“The maximum number of lines to convert and write out in one go, larger values may be a little faster but will use a lot more memory
float_format_string
format specification string to convert numbers to strings, see format_spec in python documentation
quantities
List of vector quantities, each vector quantity is one column in the text file. By default this list will be derived from the particles set. When this option is given it will override the particle set data.

By default, text files are stored with some unit information on a comment line in the header. Unfortunately, amuse cannot depend on this line to read the file. When reading a text file you always need to specify the units of every column in the file. For example, to read a file with particle positions where all positions are stored in parsec, do:

particles = io.read_set_from_file(
       'positions.txt',
       'txt',
       attribute_types = (units.parsec, units.parsec, units.parsec),
       attribute_names= ('x', 'y', 'z')
)

When writing a text file, you do not need to specify the units. If you don’t specify the units the write_set_to_file function will default to the units of the quantities being saved. For reliability and reproducibility we suggest to always specify the units and the names of the attributes to save when saving a text file:

particles = io.write_set_from_file(
       particles,
       'txt',
       attribute_types = (units.parsec, units.parsec, units.parsec),
       attribute_names= ('x', 'y', 'z')
)

The amuse text file routines are very generic and since version 6.0, you can also specify a list of vector quantities instead of a particle set:

particles = io.write_set_from_file(
       None,
       'txt',
       attribute_types = (units.parsec, units.parsec, units.parsec),
       attribute_names = ('x', 'y', 'z'),
       quantities = [
            [0.1,0.2] | units.parsec,
            [0.3,0.4] | units.parsec,
            [0.5,0.6] | units.parsec,
       ]
)

Starlab

AMUSE has support for reading and writing starlab (.dyn) files. You can specify the starlab format by entering “dyn” or “starlab” as the format:

>>> from amuse.support import io
>>> particles = io.read_set_from_file('plummer.dyn','starlab')
>>> io.write_set_to_file(particles, 'output.dyn', 'dyn')

The starlab format support sevaral options, listed below. You can use these options by adding additional keyword arguments to the read_set_from_file() or write_set_to_file() functions. For example:

>>> from amuse.support import io
>>> particles = io.read_set_from_file('plummer.dyn','starlab', must_scale = False, return_children = False)
extra_attributes
Extra attributes to store with the data set. Some formats (moste notably the amuse native format) can store extra attributes with the set in file. The ‘write_set_to_file’ function will collect all keyword arguments that do not match to an option into the extra attributes dictionary.
dynamics_time_units
The time fields in the dynamics section of a starlab file can be in Myr or in scaled units , defaults to scaled units (nbody_system.time). When set to scaled units, AMUSE will convert the units if scaling parameters are also given in the file. See the `must_scale` option to turn this scaling off.
dynamics_mass_units
The m field in the dynamics section of a starlab file can be in MSun or in scaled units , defaults to scaled units (nbody_system.mass). When set to scaled units, AMUSE will convert the units if scaling parameters are also given in the file. See the `must_scale` option to turn this scaling off
return_converter
If True also return the converter when reading a set using read_set_from_file
dynamics_length_units
The length fields in the dynamics section of a starlab file can be in parsec or in scaled units , defaults to scaled units (nbody_system.length) When set to scaled units, AMUSE will convert the units if scaling parameters are also given in the file. See the `must_scale` option to turn this scaling off.
return_children
If True returns the children of the root node, if False returns the root node (defaults to True)
must_scale
If True use the scaling values from the file, if False do not scale the stellar dynamics properties. Only used when no nbody to si converter has been set.
nbody_to_si_converter
Starlab datafiles store stellar dynamics properties in scaled nbody values, provide a converter to store si data (defaults to None). Value None means no converter, or use scaling values provided in the file

The units of the values in the star (stellar properties) section of a starlab file are always in derived S.I units (solar mass, million years, solar luminocity etc.).

The masses given in de the dynamics section of a starlab file are usually in nbody units. Some starlab tools will set the mass values in Solar mass units (for example the makemass tool will return the masses in solar mass units). To read these files you need to set the dynamics_mass_units.

> makeplummer -n 1000 > plummer1.dyn
> cat plummer1.dyn | makemass -f 1 -x -2.0 -l 0.1 -u 20 > plummer2.dyn
> cat plummer2.dyn | add_star -Q 0.5 -R 5 > plummer3.dyn
> cat plummer3.dyn | scale -s > plummer4.dyn
> cat plummer4.dyn | kira -S > plummer5.dyn

The plummer1.dyn, plummer4.dyn and plummer5.dyn files will provide masses (and all other dynamical properties) in scaled nbody units. The plummer2.dyn and plummer3.dyn files will have masses in solar masses. To read each file in AMUSE, and return the particles with S.I. units, you need to do:

>>> from amuse.support import io
>>> from amuse.units import nbody_system, units
>>> converter = nbody_system.nbody_to_si(1 | units.MSun, 1 | units.parsec)
>>> particles1 = io.read_set_from_file('plummer1.dyn','starlab', nbody_to_si_converter = converter)
>>> particles2 = io.read_set_from_file('plummer2.dyn','starlab', dynamics_mass_units = units.MSun, nbody_to_si_converter = converter)
>>> particles3 = io.read_set_from_file('plummer3.dyn','starlab', dynamics_mass_units = units.MSun, nbody_to_si_converter = converter)
>>> particles4 = io.read_set_from_file('plummer4.dyn','starlab')
>>> particles5 = io.read_set_from_file('plummer5.dyn','starlab')

Note

No nbody converter object is needed for the last files, as the scale factors given in the files will be used.

The plummer1.dyn, plummer4.dyn and plummer5.dyn can also be read in nbody units. In the following example the returned particles have dynamic attributes (mass, radius, velocity, acceleration) in nbody units:

>>> from amuse.support import io
>>> particles1 = io.read_set_from_file('plummer1.dyn','starlab')
>>> particles4 = io.read_set_from_file('plummer4.dyn','starlab', must_scale = False)
>>> particles5 = io.read_set_from_file('plummer5.dyn','starlab', must_scale = False)

NEMO

AMUSE has support for reading and writing nemo (.tsf) files. You can specify the starlab format by entering “nemo” or “tsf” as the format:

>>> from amuse.support import io
>>> particles = io.read_set_from_file('plummer.tsf','nemo')
>>> io.write_set_to_file(particles, 'output.tsf', 'tsf')

The nemo format support several options, listed below. You can use these options by adding additional keyword arguments to the read_set_from_file() or write_set_to_file() functions. For example:

>>> from amuse.support import io
>>> from amuse.units import nbody_system, units
>>> converter = nbody_system.nbody_to_si(1 | units.MSun, 1 | units.parsec)
>>> particles = io.read_set_from_file('plummer.nemo','tsf', nbody_to_si_converter = converter)
extra_attributes
Extra attributes to store with the data set. Some formats (moste notably the amuse native format) can store extra attributes with the set in file. The ‘write_set_to_file’ function will collect all keyword arguments that do not match to an option into the extra attributes dictionary.
nbody_to_si_converter
NEMO datafiles store nbody data, provide a converter to store si data (None means no converter)

Gadget

AMUSE has support for reading and writing gadget 2 files. You can specify the gadget format by entering “gadget” as the format:

>>> from amuse.support import io
>>> gas, halo, disk, bulge, stars, bndry = io.read_set_from_file('plummer.dat','gadget')
>>> io.write_set_to_file(particles, 'output.dat', 'gadget')

The gadget file format reader will return a tuple of gas, halo, disk, bulge, stars and boundary particles. This is different form other readers which only return one set. The write_set_to_file can take a particles set (will be saved as halo particles) or a tuple of gas, halo, disk, bulge, stars and boundary particles.

The gadget format support several options, listed below. You can use these options by adding additional keyword arguments to the read_set_from_file() or write_set_to_file() functions. For example (will read a gadget file with timestep information):

>>> from amuse.support import io
>>> particles = io.read_set_from_file('plummer.nemo','gadget', has_timestep = True)
is_initial_conditions_format
Set to true if the file contains initial conditions. An initial conditions gadget file contains less data.
extra_attributes
Extra attributes to store with the data set. Some formats (moste notably the amuse native format) can store extra attributes with the set in file. The ‘write_set_to_file’ function will collect all keyword arguments that do not match to an option into the extra attributes dictionary.
endianness
The endianness of the binary date stored in the file
has_rate_of_entropy_production
Set to true if the file has a block with the rate of change of the entropic function of each gas particle
ids_are_long
Set to true the ids will be written as longs in the gadget file
write_header_from
Pass a namedtuple to store non-default values in the header of the gadget file
header_struct_format
The format of the header structure of the gadget file.
return_header
Set to True to return both the particles and the header from the gadget file
has_potential_energy
Set to true if the file has a potential energy block
ids_are_keys
Set to True if the file contains correct keys. Set to False to generate the keys in amuse and prove an id attribute for the id’s in the gadget file
equal_mass_array
If filled with an array with masses > 0.0 assume equal mass for the corresponding set
convert_gadget_w_to_velocity
Set to True to convert the w=sqrt(a)*dx/dt to (comoving) velocity in comoving integrations
has_timestep
Set to true if the file has a block with the individual timestep for each particle.
has_acceleration
Set to true if the file has a acceleration block

Table Of Contents

Previous topic

Support code for the community code interfaces

Next topic

Reporting during a run

This Page