Skip to main content
`Ctrl`+`K`

# Lazy loading

Data can be loaded lazily by using `lazy=True`, however not all formats are supported. The data will be loaded as a dask array instead of a numpy array.

    >>> from rsciio import msa
    >>> d = msa.file_reader("file.mrc", lazy=True)
    >>> d["data"]
    dask.array<array, shape=(10, 20, 30), dtype=int64, chunksize=(10, 20, 30), chunktype=numpy.ndarray>

## Chunks

Depending on the intended processing after loading the data, it may be necessary to define the chunking manually to control the memory usage or compute distribution. The chunking can also be specified as follow using the `chunks` parameter:

    >>> s = hs.load("file.mrc", lazy=True, chunks=(5, 10, 10))

## Memory mapping

Binary file formats are loaded lazily using memory mapping and are compatible with the dask distributed scheduler. This implementation uses an approach similar to that described in the dask documentation on memory mapping - see the `memmap_distributed()` function for more information.

## Distributed Loading

Not all formats are compatible with the dask distributed scheduler. See the last columns of the supported formats table to know which reader are supported.

In almost all cases the `memmap_distributed()` function can be dropped in-place of the `numpy.memmap` function. It also now supports the `positions` parameter which is different from the equivalent numpy function. The `positions` parameter is a numpy array of positions which maps some arbitrary scan positions to a grid. This is useful for loading arbitrary scan positions from a file. The `positions` parameter does require that the data is chunked only in the navigation axis.