Skip to main content
`Ctrl`+`K`

# C-Blosc2

## A fast, compressed and persistent data store library for C

Author:

  Blosc Development Team

Contact:

  blosc@blosc.org

URL:

  https://www.blosc.org

## What is it?

Blosc2 is a high-performance compressor and data format optimized for binary data, including numerical arrays, tensors, and other structured formats. It provides a flexible framework of codecs and filters, enabling developers to balance compression speed and ratio for specific use cases, from high-throughput data pipelines to persistent storage. As the successor to the original Blosc library (released in 2010), Blosc2 is built on a mature foundation and is integrated into many popular scientific computing libraries, such as PyTables, h5py, and Zarr.

C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. However, the reverse thing is generally not true for the format; buffers generated with C-Blosc2 are not format-compatible with C-Blosc1 (i.e. forward compatibility is not supported). In case you want to ensure full API compatibility with C-Blosc1 API, define the BLOSC1_COMPAT symbol.

See a 3-minute introductory video to Blosc2.

## Blosc2 NDim: an N-Dimensional store

One of more exciting additions in C-Blosc2 is the Blosc2 NDim layer (or B2ND for short), which allows creating and reading n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, that allows slicing and dicing arbitrary large and compressed data in a more fine-grained way:

To whet your appetite, here is how the NDArray object in the Python wrapper performs on getting slices orthogonal to the different axis of a 4-dim dataset:

We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro

We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful.

## New features in C-Blosc2

Right now, the next features are already implemented (although they may require some refactoring or more tests):

- 64-bit containers: the first-class container in C-Blosc2 is the super-chunk or, for brevity, schunk, that is made by smaller chunks which are essentially C-Blosc1 32-bit containers. The super-chunk can be backed or not by another container which is called a frame (see later).

- More filters: besides shuffle and bitshuffle already present in C-Blosc1, C-Blosc2 already implements:

  - delta: the stored blocks inside a chunk are diff’ed with respect to first block in the chunk. The idea is that, in some situations, the diff will have more zeros than the original data, leading to better compression.

  - trunc_prec: it zeroes the least significant bits of the mantissa of float32 and float64 types. When combined with the shuffle or bitshuffle filter, this leads to more contiguous zeros, which are compressed better.

- A filter pipeline: the different filters can be pipelined so that the output of one can be the input for the other. A possible example is a delta followed by shuffle, or as described above, trunc_prec followed by bitshuffle.

- Prefilters: allows applying user-defined C callbacks prior to the filter pipeline during compression. See test_prefilter.c for an example of use.

- Postfilters: allows applying user-defined C callbacks after the filter pipeline during decompression. The combination of prefilters and postfilters could be interesting for supporting e.g. encryption (via prefilters) and decryption (via postfilters). Also, a postfilter alone can be used to produce on-the-fly computation based on existing data (or other metadata, like e.g. coordinates). See test_postfilter.c for an example of use.

- SIMD support for ARM (NEON): this allows for faster operation on ARM architectures. Only shuffle is supported right now, but the idea is to implement bitshuffle for NEON too. Thanks to Lucian Marc.

- SIMD support for PowerPC (ALTIVEC): this allows for faster operation on PowerPC architectures. Both shuffle and bitshuffle are supported; however, this has been done via a transparent mapping from SSE2 into ALTIVEC emulation in GCC 8, so performance could be better (but still, it is already a nice improvement over native C code; see PR Blosc/c-blosc2#59 for details). Thanks to Jerome Kieffer and ESRF for sponsoring the Blosc team in doing this task.

- Dictionaries: when a block is going to be compressed, C-Blosc2 can use a previously made dictionary (stored in the header of the super-chunk) for compressing all the blocks that are part of the chunks. This usually improves the compression ratio, as well as the decompression speed, at the expense of a (small) overhead in compression speed. Currently, it is only supported in the zstd codec, but would be nice to extend it to lz4 and blosclz at least.

- Contiguous frames: allow to store super-chunks contiguously, either on-disk or in-memory. When a super-chunk is backed by a frame, instead of storing all the chunks sparsely in-memory, they are serialized inside the frame container. The frame can be stored on-disk too, meaning that persistence of super-chunks is supported.

- Sparse frames: each chunk in a super-chunk, as well as the metadata, are stored separately. This allows for more efficient updates than in frames (i.e. avoiding ‘holes’ in monolithic files).

- Partial chunk reads: there is support for reading just part of a chunk, so avoiding reading the whole thing and then discarding the unnecessary data (which is a waste of resources).

- Parallel chunk reads: when several blocks of a chunk are to be read, this is done in parallel by the decompressing machinery. That means that every thread is responsible to read, post-filter and decompress a block by itself, leading to an efficient overlap of I/O and CPU usage that optimizes reads to a maximum.

- Meta-layers: optionally, the user can add meta-data for different uses and in different layers. For example, one may think on providing a meta-layer for NumPy so that most of the meta-data for it is stored in a meta-layer; then, one can place another meta-layer on top of the latter for adding more high-level info if desired (e.g. geo-spatial, meteorological…).

- Variable length meta-layers: the user may want to add variable-length meta information that can be potentially very large (up to 2 GB). The regular meta-layer described above is very quick to read, but meant to store fixed-length and relatively small meta information. Variable length metalayers are stored in the trailer of a frame, whereas regular meta-layers are in the header.

- Efficient support for special values: large sequences of repeated values can be represented with an efficient, simple and fast run-length representation, without the need to use regular codecs. With that, chunks or super-chunks with values that are the same (zeros, NaNs or any value in general) can be built in constant time, regardless of the size. This can be useful in situations where a lot of zeros (or NaNs) need to be stored (e.g. sparse matrices).

- Nice markup for documentation: we are currently using a combination of Sphinx + Doxygen + Breathe for documenting the C-API. See https://www.blosc.org/c-blosc2/c-blosc2.html. Thanks to Alberto Sabater and Aleix Alcacer for contributing the support for this.

- Plugin capabilities for filters and codecs: we have a plugin register capability inplace so that the info about the new filters and codecs can be persisted and transmitted to different machines. Thanks to the NumFOCUS foundation for providing a grant for doing this.

- Centralized plugin repository: we have implemented a centralized repository so that people can send their plugins (using the existing machinery) to the Blosc2 team. If the plugins fulfill a series of requirements, they will be officially accepted, and distributed within the library. Thanks to NumFOCUS foundation for providing a grant for doing this. See https://www.blosc.org/posts/registering-plugins/.

- Support for lossy codecs: besides supporting the trunc_prec filter (described above), we also offer support for zfp, a codec that is specifically meant for lossy compression of multidimensional floating point data. For details on how use it, see Blosc/c-blosc2. Support for more lossy codecs may come in the future.

- Pluggable tuning capabilities: this will allow users with different needs to define an interface so as to better tune different parameters like the codec, the compression level, the filters to use, the blocksize or the shuffle size. Thanks to ironArray for sponsoring us in doing this.

- Support for I/O plugins: so that users can extend the I/O capabilities beyond the current filesystem support. Things like use databases or S3 interfaces should be possible by implementing these interfaces. Thanks to ironArray for sponsoring us in doing this.

- Python wrapper: we have a preliminary wrapper in the works. You can have a look at our ongoing efforts in the python-blosc2 repo. Thanks to the Python Software Foundation for providing a grant for doing this.

- Security: we are actively using the OSS-Fuzz and ClusterFuzz for uncovering programming errors in C-Blosc2. Thanks to Google for sponsoring us in doing this.

More info about the improved capabilities of C-Blosc2 can be found in this paper. Please, cite it if you use C-Blosc2 in your research!

## Open format

The Blosc2 format is open and fully documented.

The format specs are defined in less than 4000 words, so they should be easy to read and understand. In our opinion, this is critical for the long-term success of the library, as it allows for third-party implementations of the format, and also for the users to understand what is going on under the hood.

## Python wrapper

We are officially supporting (thanks to the Python Software Foundation) a Python wrapper for Blosc2. It supports all the features of the predecessor python-blosc package plus most of the bells and whistles of C-Blosc2, like 64-bit and multidimensional containers. As a bonus, the python-blosc2 package comes with wheels and binary versions of the C-Blosc2 libraries, so anyone, even non-Python users can install C-Blosc2 binaries easily with:

    pip install blosc2

## Compiling the C-Blosc2 library with CMake

Blosc can be built, tested and installed using CMake. The following procedure describes a typical CMake build.

Create the build directory inside the sources and move into it:

    git clone https://github.com/Blosc/c-blosc2
    cd c-blosc2
    mkdir build
    cd build

Now run CMake configuration and optionally specify the installation directory (e.g. ‘/usr’ or ‘/usr/local’):

    cmake -DCMAKE_INSTALL_PREFIX=your_install_prefix_directory ..

CMake allows to configure Blosc in many different ways, like preferring internal or external sources for compressors or enabling/disabling them. Please note that configuration can also be performed using UI tools provided by CMake (ccmake or cmake-gui):

    ccmake ..      # run a curses-based interface
    cmake-gui ..   # run a graphical interface

Build, test and install Blosc:

    cmake --build .
    ctest
    cmake --build . --target install

The static and dynamic version of the Blosc library, together with header files, will be installed into the specified CMAKE_INSTALL_PREFIX.

Once you have compiled your Blosc library, you can easily link your apps with it as shown in the examples/ directory.

### Handling support for codecs (LZ4, LZ4HC, Zstd, Zlib)

C-Blosc2 comes with full sources for LZ4, LZ4HC, Zstd, and Zlib and in general, you should not worry about not having (or CMake not finding) the libraries in your system because by default the included sources will be automatically compiled and included in the C-Blosc2 library. This means that you can be confident in having a complete support for all these codecs in all the official Blosc deployments. Of course, if you consider this is too bloated, you can exclude support for some of them.

For example, let’s suppose that you want to disable support for Zstd:

    cmake -DDEACTIVATE_ZSTD=ON ..

Or, you may want to use a codec in an external library already in the system:

    cmake -DPREFER_EXTERNAL_LZ4=ON ..

### Supported platforms

C-Blosc2 is meant to support all platforms where a C99 compliant C compiler can be found. The ones that are mostly tested are Intel (Linux, Mac OSX and Windows), ARM (Linux, Mac), and PowerPC (Linux). More on ARM support in README_ARM.rst.

For Windows, you will need at least VS2015 or higher on x86 and x64 targets (i.e. ARM is not supported on Windows).

For Mac OSX, make sure that you have the command line developer tools available. You can always install them with:

    xcode-select --install

For Mac OSX on arm64 architecture, you may want to compile it like this:

    CC="clang -arch arm64" cmake ..

### Display error messages

By default error messages are disabled. To display them, you just need to activate the Blosc tracing machinery by setting the `BLOSC_TRACE` environment variable.

## Contributing

If you want to collaborate in this development you are welcome. We need help in the different areas listed at the ROADMAP; also, be sure to read our DEVELOPING-GUIDE and our Code of Conduct. Blosc is distributed using the BSD license.

## Twitter feed

Follow @Blosc2 so as to get informed about the latest developments.

## Citing Blosc

You can cite our work on the different libraries under the Blosc umbrella as:

    @ONLINE{blosc,
      author = {{Blosc Development Team}},
      title = "{A fast, compressed and persistent data store library}",
      year = {2009-2025},
      note = {https://blosc.org}
    }

## Acknowledgments

See THANKS document.

Compress Better, Compute Bigger

– Blosc Development Team

## Contents

- API Reference
  - Utility variables
  - Utility functions
  - Blosc1 API
  - Context
  - Plugins
  - Super-chunk
  - Metalayers
  - Blosc2 NDim
- Format
  - B2ND Format
  - B2ND Metalayer Format
  - Blosc2 Sparse Frame Format
  - Blosc2 Contiguous Frame Format
  - Blosc/Blosc2 Chunk Format
  - Extensions for Blosc2 Filenames
- Development
  - Contributing guidelines
  - Releasing a version
  - Roadmap to 3.0