# Introduction

## Getting Started

Most of the functionality that one will use is contained in the main module

    import hdf5storage

Lower level functionality needed mostly for extending this package to work with more datatypes are in its submodules.

The main functions in this module are `write()` and `read()` which write a single Python variable to an HDF5 file or read the specified contents at one location in an HDF5 file and convert to Python types.

HDF5 files are structured much like a Unix filesystem, so everything can be referenced with a POSIX style path, which look like `'/pyth/hf'`. Unlike a Windows path, back slashes (`'/'`) are used as directory separators instead of forward slashes (`'\'`) and the base of the file system is just `'/'` instead of something like `'C:\'`. In the language of HDF5, what we call directories and files in filesystems are called groups and datasets.

`write()` has many options for controlling how the data is stored, and what metadata is stored, but we can ignore that for now. If we have a variable named `foo` that we want to write to an HDF5 file named `data.h5`, we would write it by

    hdf5storage.write(foo, path='/foo', filename='data.h5')

And then we can read it back from the file with the `read()` function, which returns the read data. Here, we will put the data we read back into the variable `bar`

    bar = hdf5storage.read(path='/foo', filename='data.h5')

## Writing And Reading Several Python Variables at Once

To write and read more than one Python variable, one could use `write()` and `read()` for each variable individually. This can incur a major performance penalty, especially for large HDF5 files, since each call opens and closes the HDF5 file (sometimes more than once).

Version `0.1.10` added a way to do this without incuring this performance penalty by adding two new functions: `writes()` and `reads()`.

They can write and read more than one Python variable at once, though they can still work with a single variable. In fact, `write()` and `read()` are now wrappers around them. `savemat()` and `loadmat()` currently use them for the improved performance.

New in version 0.1.10: Ability to write and read more than one Python variable at a time without opening and closing the HDF5 file each time.

## Main Options Controlling Writing/Reading Data

There are many individual options that control how data is written and read to/from file. These can be set by passing an `Options` object to `write()` and `read()` by

    options = hdf5storage.Options(...)
    hdf5storage.write(... , options=options)
    hdf5storage.read(... , options=options)

or passing the individual keyword arguments used by the `Options` constructor to `write()` and `read()`. The two methods cannot be mixed (the functions will give precedence to the given `Options` object).

Note

Functions in the various submodules only support the `Options` object method of passing options.

The two main options are `Options.store_python_metadata` and `Options.matlab_compatible`. A more minor option is `Options.oned_as`.

New in version 0.1.9: Support for the transparent compression of data has been added. It is enabled by default, compressing all python objects resulting in HDF5 Datasets larger than 16 KB with the GZIP/Deflate algorithm.

### store_python_metadata

`bool`

Setting this options causes metadata to be written so that the written objects can be read back into Python accurately. As HDF5 does not natively support many Python data types (essentially only Numpy types), most Python data types have to be converted before being written. If metadata isn’t also written, the data cannot be read back to its original form and will instead be read back as the Python type most closely resembling how it is stored, which will be a Numpy type of some sort.

### matlab_compatible

`bool`

Setting this option causes the writing of HDF5 files be done in a way compatible with MATLAB v7.3 MAT files. This consists of writing some file metadata so that MATLAB recognizes the file, adding specific metadata to every stored object so that MATLAB recognizes them, and transforming the data to be in the form that MATLAB expects for certain types (for example, MATLAB expects everything to be at least a 2D array and strings to be stored in UTF-16 but with no doublets).

Note

There are many individual small options in the `Options` class that this option sets to specific values. Setting `matlab_compatible` automatically sets them, while changing their values to something else automatically turns `matlab_compatible` off.

### action_for_matlab_incompatible

{`'ignore'`, `'discard'`, `'error'`}

The action to perform when doing MATLAB compatibility (`matlab_compatible == True`) but a type being written is not MATLAB compatible. The actions are to write the data anyways (‘ignore’), don’t write the incompatible data (‘discard’), or throw a `lowlevel.TypeNotMatlabCompatibleError` exception. The default is ‘error’.

### oned_as

{‘row’, ‘column’}

This option is only actually relevant when `matlab_compatible == True`. MATLAB only supports 2D and higher dimensionality arrays, but Numpy supports 1D arrays. So, 1D arrays have to be made 2 dimensional making them either into row vectors or column vectors. This option sets which they become when imported into MATLAB.

### compress

New in version 0.1.9.

`bool`

Whether to use compression when writing data. Enabled (`True`) by default. See Compression for more information.

## Convenience Functions for MATLAB MAT Files

Two functions are provided for reading and writing to MATLAB MAT files in a convenient way. They are `savemat()` and `loadmat()`, which are modelled after the SciPy functions of the same name (`scipy.io.savemat()` and `scipy.io.loadmat()`), which work with non-HDF5 based MAT files. They take not only the same options, but dispatch calls automatically to the SciPy versions when instructed to write to a non-HDF5 based MAT file, or read a MAT file that is not HDF5 based. SciPy must be installed to take advantage of this functionality.

`savemat()` takes a `dict` having data (values) and the names to give each piece of data (keys), and writes them to a MATLAB compatible MAT file. The format keyword sets the MAT file format, with `'7.3'` being the HDF5 based format supported by this package and `'5'` and `'4'` being the non HDF5 based formats supported by SciPy. If you want the data to be able to be read accurately back into Python, you should set `store_python_metadata=True`. Writing a couple variables to a file looks like

    hdf5storage.savemat('data.mat', {'foo': 2.3, 'bar': (1+2j)}, format='7.3', oned_as='column', store_python_metadata=True)

Then, to read variables back, we can either explicitly name the variables we want

    out = hdf5storage.loadmat('data.mat', variable_names=['foo', 'bar'])

or grab all variables by either not giving the variable_names option or setting it to `None`.

    out = hdf5storage.loadmat('data.mat')

## Example: Write And Readback Including Different Metadata

### Making The Data

Make a `dict` containing many different types in it that we want to store to disk in an HDF5 file. The initialization method depends on the Python version.

#### Python 3

The `dict` keys must be `str` (the unicode string type).

    >>> import numpy as np
    >>> import hdf5storage
    >>> a = {'a': True,
    ...      'b': None,
    ...      'c': 2,
    ...      'd': -3.2,
    ...      'e': (1-2.3j),
    ...      'f': 'hello',
    ...      'g': b'goodbye',
    ...      'h': ['list', 'of', 'stuff', [30, 2.3]],
    ...      'i': np.zeros(shape=(2,), dtype=[('bi', 'uint8')]),
    ...      'j':{'aa': np.bool_(False),
    ...           'bb': np.uint8(4),
    ...           'cc': np.uint32([70, 8]),
    ...           'dd': np.int32([]),
    ...           'ee': np.float32([[3.3], [5.3e3]]),
    ...           'ff': np.complex128([[3.4, 3], [9+2j, 0]]),
    ...           'gg': np.array(['one', 'two', 'three'], dtype='str'),
    ...           'hh': np.bytes_(b'how many?'),
    ...           'ii': np.object_(['text', np.int8([1, -3, 0])])}}

#### Python 2

The same thing but in Python 2 where the `dict` keys must be `unicode`. The other datatypes are translated from the Python 3 example appropriately. The rest of the examples on this page are run identically in Python 2 and 3, but the outputs are listed as is returned in Python 3.

    >>> import numpy as np
    >>> import hdf5storage
    >>> a = {u'a': True,
    ...      u'b': None,
    ...      u'c': 2,
    ...      u'd': -3.2,
    ...      u'e': (1-2.3j),
    ...      u'f': u'hello',
    ...      u'g': 'goodbye',
    ...      u'h': [u'list', u'of', u'stuff', [30, 2.3]],
    ...      u'i': np.zeros(shape=(2,), dtype=[('bi', 'uint8')]),
    ...      u'j':{u'aa': np.bool_(False),
    ...            u'bb': np.uint8(4),
    ...            u'cc': np.uint32([70, 8]),
    ...            u'dd': np.int32([]),
    ...            u'ee': np.float32([[3.3], [5.3e3]]),
    ...            u'ff': np.complex128([[3.4, 3], [9+2j, 0]]),
    ...            u'gg': np.array([u'one', u'two', u'three'], dtype='unicode'),
    ...            u'hh': np.str_('how many?'),
    ...            u'ii': np.object_([u'text', np.int8([1, -3, 0])])}}

### Using No Metadata

Write it to a file at the `'/a'` directory, but include no Python or MATLAB metadata. Then, read it back and notice that many objects come back quite different from what was written. Namely, everything was converted to Numpy types. This even included the dictionaries which were converted to structured `np.ndarray``s. This happens because all other types (other than ``dict`) must be converted to these types before being written to the HDF5 file, and without metadata, the conversion cannot be reversed (while `dict` isn’t converted, it has the same form and thus cannot be extracted reversibly).

    >>> hdf5storage.write(data=a, path='/a', filename='data.h5',
    ...                   store_python_metadata=False,
    ...                   matlab_compatible=False)
    >>> hdf5storage.read(path='/a', filename='data.h5')
    array([ (True,
             [],
             2,
             -3.2,
             (1-2.3j),
             b'hello',
             b'goodbye',
             [array(b'list', dtype='|S4'),
              array(b'of', dtype='|S2'),
              array(b'stuff', dtype='|S5'),
              array([array(30), array(2.3)], dtype=object)],
             [(0,), (0,)],
             [(False,
               4,
               array([70,  8], dtype=uint32),
               array([], dtype=int32),
               array([[  3.29999995e+00], [  5.30000000e+03]], dtype=float32),
               array([[ 3.4+0.j,  3.0+0.j], [ 9.0+2.j,  0.0+0.j]]),
               array([111, 110, 101,   0,   0, 116, 119, 111,   0,   0, 116, 104, 114,
                      101, 101], dtype=uint32),
               b'how many?',
               array([array(b'text', dtype='|S4'),
                      array([ 1, -3,  0], dtype=int8)],
                     dtype=object))])],
          dtype=[('a', '?'),
                 ('b', '<f8', (0,)),
                 ('c', '<i8'),
                 ('d', '<f8'),
                 ('e', '<c16'),
                 ('f', 'S5'),
                 ('g', 'S7'), ('h', 'O', (4,)),
                 ('i', [('bi', 'u1')], (2,)),
                 ('j', [('aa', '?'),
                        ('bb', 'u1'),
                        ('cc', '<u4', (2,)),
                        ('dd', '<i4', (0,)),
                        ('ee', '<f4', (2, 1)),
                        ('ff', '<c16', (2, 2)),
                        ('gg', '<u4', (15,)),
                        ('hh', 'S9'),
                        ('ii', 'O', (2,))],
                  (1,))])

### Including Python Metadata

Do the same thing, but now include Python metadata (`store_python_metadata == True`). This time, everything is read back the same (or at least, it should) as it was written.

    >>> hdf5storage.write(data=a, path='/a', filename='data_typeinfo.h5',
    ...                   store_python_metadata=True,
    ...                   matlab_compatible=False)
    >>> hdf5storage.read(path='/a', filename='data_typeinfo.h5')
    {'a': True,
     'b': None,
     'c': 2,
     'd': -3.2,
     'e': (1-2.3j),
     'f': 'hello',
     'g': b'goodbye',
     'h': ['list', 'of', 'stuff', [30, 2.3]],
     'i': array([(0,), (0,)],
          dtype=[('bi', 'u1')]),
     'j': {'aa': False,
      'bb': 4,
      'cc': array([70,  8], dtype=uint32),
      'dd': array([], dtype=int32),
      'ee': array([[  3.29999995e+00],
           [  5.30000000e+03]], dtype=float32),
      'ff': array([[ 3.4+0.j,  3.0+0.j],
           [ 9.0+2.j,  0.0+0.j]]),
      'gg': array(['one', 'two', 'three'],
          dtype='<U5'),
      'hh': b'how many?',
      'ii': array(['text', array([ 1, -3,  0], dtype=int8)], dtype=object)}}

### Including MATLAB Metadata

Do the same thing, but this time including only MATLAB metadata (`matlab_compatible == True`). This time, the data that is read back is different from what was written, but in a different way than when no metadata was used. The biggest differences are that everything was turned into an at least 2D array, all arrays are transposed, and all string types got converted to `numpy.str_`. This happens because MATLAB can only work with 2D and higher arrays, uses Fortran array ordering instead of C ordering like Python does, and strings are stored in a subset of UTF-16 (no doublets) in the version 7.3 MAT files.

    >>> hdf5storage.write(data=a, path='/a', filename='data.mat',
    ...                   store_python_metadata=False,
    ...                   matlab_compatible=True)
    >>> hdf5storage.read(path='/a', filename='data.mat')
    array([ ([[True]],
             [[]],
             [[2]],
             [[-3.2]],
             [[(1-2.3j)]],
             [['hello']],
             [['goodbye']],
             [[array([['list']], dtype='<U4'),
               array([['of']], dtype='<U2'),
               array([['stuff']], dtype='<U5'),
               array([[array([[30]]), array([[ 2.3]])]], dtype=object)]],
             [[(array([[0]], dtype=uint8),)],
              [(array([[0]], dtype=uint8),)]],
             [(array([[False]], dtype=bool),
               array([[4]], dtype=uint8),
               array([[70,  8]], dtype=uint32),
               array([], shape=(1, 0), dtype=int32),
               array([[  3.29999995e+00], [  5.30000000e+03]], dtype=float32),
               array([[ 3.4+0.j,  3.0+0.j], [ 9.0+2.j,  0.0+0.j]]),
               array([['one\x00\x00two\x00\x00three']], dtype='<U15'),
               array([['how many?']], dtype='<U9'),
               array([[array([['text']], dtype='<U4'),
                       array([[ 1, -3,  0]], dtype=int8)]], dtype=object))])],
          dtype=[('a', '?', (1, 1)),
                 ('b', '<f8', (1, 0)),
                 ('c', '<i8', (1, 1)),
                 ('d', '<f8', (1, 1)),
                 ('e', '<c16', (1, 1)),
                 ('f', '<U5', (1, 1)),
                 ('g', '<U7', (1, 1)),
                 ('h', 'O', (1, 4)),
                 ('i', [('bi', 'u1', (1, 1))], (2, 1)),
                 ('j', [('aa', '?', (1, 1)),
                        ('bb', 'u1', (1, 1)),
                        ('cc', '<u4', (1, 2)),
                        ('dd', '<i4', (1, 0)),
                        ('ee', '<f4', (2, 1)),
                        ('ff', '<c16', (2, 2)),
                        ('gg', '<U15', (1, 1)),
                        ('hh', '<U9', (1, 1)),
                        ('ii', 'O', (1, 2))],
                  (1,))])

### Including both Python And MATLAB Metadata

Do the same thing, but now include both Python metadata (`store_python_metadata == True`) and MATLAB metadata (`matlab_compatible == True`). This time, everything is read back the same (or at least, it should) as it was written. The Python metadata allows the transformations done by making the stored data MATLAB compatible reversible.

    >>> hdf5storage.write(data=a, path='/a', filename='data_typeinfo.mat',
    ...                   store_python_metadata=True,
    ...                   matlab_compatible=True)
    >>> hdf5storage.read(path='/a', filename='data_typeinfo.mat')
    {'a': True,
     'b': None,
     'c': 2,
     'd': -3.2,
     'e': (1-2.3j),
     'f': 'hello',
     'g': b'goodbye',
     'h': ['list', 'of', 'stuff', [30, 2.3]],
     'i': array([(0,), (0,)],
          dtype=[('bi', 'u1')]),
     'j': {'aa': False,
      'bb': 4,
      'cc': array([70,  8], dtype=uint32),
      'dd': array([], dtype=int32),
      'ee': array([[  3.29999995e+00],
           [  5.30000000e+03]], dtype=float32),
      'ff': array([[ 3.4+0.j,  3.0+0.j],
           [ 9.0+2.j,  0.0+0.j]]),
      'gg': array(['one', 'two', 'three'],
          dtype='<U5'),
      'hh': b'how many?',
      'ii': array(['text', array([ 1, -3,  0], dtype=int8)], dtype=object)}}