Skip to main content
`Ctrl`+`K`

# ZSpy - HyperSpy’s Zarr Specification

Note

To read this format, the optional dependency `zarr` is required.

Similarly to the hspy format, the `.zspy` format guarantees that no information will be lost in the writing process and that supports saving data of arbitrary dimensions. It is based on the Zarr project. Which exists as a drop in replacement for hdf5 with the intention to fix some of the speed and scaling issues with the hdf5 format and is therefore suitable for saving . Example using HyperSpy:

    >>> import hyperspy.api as hs
    >>> s = hs.signals.BaseSignal([0])
    >>> s.save('test.zspy') # will save in nested directory
    >>> hs.load('test.zspy') # loads the directory

When saving to zspy, all supported objects in the signal’s `hyperspy.api.signals.BaseSignal.metadata` is stored. This includes lists, tuples and signals. Please note that in order to increase saving efficiency and speed, if possible, the inner-most structures are converted to numpy arrays when saved. This procedure homogenizes any types of the objects inside, most notably casting numbers as strings if any other strings are present:

## Using different Zarr stores

By default, a `zarr.storage.NestedDirectoryStore` is used, but other zarr store can be used when saving to file and this can be done by using one of these two approaches:

- pass `"local"` or `"zip"` to the `store_type` parameter to use a `zarr.storage.NestedDirectoryStore` or a `zarr.storage.ZipStore` zarr store, respectively;

- pass an instance of `zarr.storage` to the `filename` parameter.

Note

While the `ZipStore` is convenient to save as a single file, it has limitations:

- The `ZipStore` currently doesn’t support overwriting and will raise a `NotImplementedError` when writing to an existing file.

- It doesn’t support writing from multiple processes at the same time.

Example of saving a signal using the `ZipStore`:

    >>> import zarr
    >>> signal.save('test.zspy', store_type="zip") # saved using ZipStore

To load this file again, RosettaSciIO will automatically detect that it is a zip file and load it using the `ZipStore`:

    >>> import zarr
    >>> s = hs.load('test.zspy') # load from ZipStore

If other zarr stores are used, it will be necessary to specify the store type when loading the file:

    >>> import zarr
    >>> store = zarr.storage.ZipStore('test.zspy')  # or the store the data was saved with
    >>> s = hs.load(store) # load using the given store

## API functions

rsciio.zspy.file_reader(filename, lazy=False, **kwds)

  Read data from zspy files saved with the HyperSpy zarr format specification.

  Parameters:
    filename `python:str`, `pathlib.Path`

      Filename of the file to read or corresponding pathlib.Path.

    lazy bool, default=False

      Whether to open the file lazily or not. The file will stay open until closed in `compute()` or closed manually. `get_file_handle()` can be used to access the file handler and close it manually.

    **kwds `python:dict`, optional

      Pass keyword arguments to the `zarr.convenience.open()` function.

  Returns:
    `python:list` of `python:dict`

      List of dictionaries containing the following fields:

      - ‘data’ – multidimensional `numpy.ndarray` or `dask.array.Array`

      - ‘axes’ – list of dictionaries describing the axes containing the fields ‘name’, ‘units’, ‘index_in_array’, and either ‘size’, ‘offset’, and ‘scale’ or a numpy array ‘axis’ containing the full axes vector

      - ‘metadata’ – dictionary containing the parsed metadata

      - ‘original_metadata’ – dictionary containing the full metadata tree from the input file

      When the file contains several datasets, each dataset will be loaded as separate dictionary.

rsciio.zspy.file_writer(filename, signal, chunks=None, compressor=None, close_file=True, write_dataset=True, store_type=None, show_progressbar=True, **kwds)

  Write data to HyperSpy’s zarr format.

  Parameters:
    filename `python:str`, `pathlib.Path`

      Filename of the file to write to or corresponding pathlib.Path.

    signal `python:dict`

      Dictionary containing the signal object. Should contain the following fields:

      - ‘data’ – multidimensional numpy array

      - ‘axes’ – list of dictionaries describing the axes containing the fields ‘name’, ‘units’, ‘index_in_array’, and either ‘size’, ‘offset’, and ‘scale’ or a numpy array ‘axis’ containing the full axes vector

      - ‘metadata’ – dictionary containing the metadata tree

    chunks `python:tuple` of `python:int` or `python:None`, default=None

      Define the chunking used for saving the dataset. If `None`, calculates chunks for the signal, with preferably at least one chunk per signal space.

    compressor `numcodecs.abc.Codec` or `python:None`, default=None

      A compressor can be passed to the save function to compress the data efficiently, see Numcodecs codec. If None, use a Blosc compressor.

    close_file bool, default=True

      Close the file after writing. Only relevant for some zarr storages (`zarr.storage.ZipStore`, `zarr.storage.DBMStore`) requiring store to flush data to disk. If `False`, doesn’t close the file after writing. The file should not be closed if the data needs to be accessed lazily after saving.

    write_dataset bool, default=True

      If `False`, doesn’t write the dataset when writing the file. This can be useful to overwrite signal attributes only (for example `axes_manager`) without having to write the whole dataset, which can take time.

    store_type `python:str`, “local” or “zip” or `python:None`

      If “local”, uses a `zarr.storage.NestedDirectoryStore` to save the file in a local directory. If “zip”, uses a `zarr.storage.ZipStore` to save the file in a zip archive. If `None`, the default store is used (`NestedDirectoryStore`) is used. Specifying this parameter is incompatible with passing an instance of a zarr store to the `filename` parameter. Default is None.

    show_progressbar bool, default=True

      Whether to show the progressbar or not.

    **kwds

      The keyword arguments are passed to the `zarr.hierarchy.Group.require_dataset()` function.

  Examples

      >>> from numcodecs import Blosc
      >>> compressor = Blosc(cname='zstd', clevel=1, shuffle=Blosc.SHUFFLE) # Used by default
      >>> file_writer("test.zspy", signal_dict, compressor=compressor) # will save with Blosc compression

  Using a `zarr.storage.ZipStore` store:

      >>> file_writer("test.zspy", signal_dict, store_type="zip")

Note

Lazy operations are often i-o bound. Reading and writing the data creates a bottle neck in processes due to the slow read write speed of many hard disks. In these cases, compressing your data is often beneficial to the speed of some operations. Compression speeds up the process as there is less to read/write with the trade off of slightly more computational work on the CPU.