# Direct Writes (and Reads)

The purpose of direct writes is to enable an application to write data that is already compressed in memory directly to an HDF5 file without first uncompressing it so the filter can then turn around and compress it during write. However, once data is written to the file with a direct write, consumers must still be able to read it without concern for how the producer wrote it.

Doing this requires the use of an advanced HDF5 function for direct writes.

At present, we demonstrate only minimal functionality here using single chunking, where the chunk size is chosen to match the size of the entire dataset. To see an example of code that does this, have a look at…

     1    if (zfparr>0 && zfpmode==1 && rate>0)
     2    {
     3        int            dims[] = {38, 128};
     4       /*int      chunk_dims[] = {19, 34};*/
     5        int      chunk_dims[] = {38, 128};
     6        hsize_t       hdims[] = {38, 128};
     7       /*hsize_t hchunk_dims[] = {19, 34};*/
     8        hsize_t hchunk_dims[] = {38, 128};
     9        hsize_t hchunk_off[] = {0, 0};
    10#if defined(ZFP_LIB_VERSION) && ZFP_LIB_VERSION<=0x055
    11        cfp_array2d *origarr;
    12#else
    13        cfp_array2d origarr;
    14#endif
    15
    16        /* Create the array data */
    17        buf = gen_random_correlated_array(TYPDBL, 2, dims, 0, 0);
    18
    19        /* Instantiate a cfp array */
    20        origarr = cfp.array2d.ctor(dims[1], dims[0], rate, buf, 0);
    21        cfp.array2d.flush_cache(origarr);
    22
    23        cpid = setup_filter(2, hchunk_dims, 1, rate, acc, prec, minbits, maxbits, maxprec, minexp);
    24
    25        if (0 > (sid = H5Screate_simple(2, hdims, 0))) SET_ERROR(H5Screate_simple);
    26
    27        /* write the data WITHOUT compression */
    28        if (0 > (dsid = H5Dcreate(fid, "zfparr_original", H5T_NATIVE_DOUBLE, sid, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT))) SET_ERROR(H5Dcreate);
    29        if (0 > H5Dwrite(dsid, H5T_NATIVE_DOUBLE, H5S_ALL, H5S_ALL, H5P_DEFAULT, buf)) SET_ERROR(H5Dwrite);
    30        if (0 > H5Dclose(dsid)) SET_ERROR(H5Dclose);
    31
    32        /* write the data with compression via the filter */
    33        if (0 > (dsid = H5Dcreate(fid, "zfparr_compressed", H5T_NATIVE_DOUBLE, sid, H5P_DEFAULT, cpid, H5P_DEFAULT))) SET_ERROR(H5Dcreate);
    34        if (0 > H5Dwrite(dsid, H5T_NATIVE_DOUBLE, H5S_ALL, H5S_ALL, H5P_DEFAULT, buf)) SET_ERROR(H5Dwrite);
    35        if (0 > H5Dclose(dsid)) SET_ERROR(H5Dclose);
    36
    37        /* write the data direct from compressed array using H5Dwrite_chunk calls */
    38        if (0 > (dsid = H5Dcreate(fid, "zfparr_direct", H5T_NATIVE_DOUBLE, sid, H5P_DEFAULT, cpid, H5P_DEFAULT))) SET_ERROR(H5Dcreate);
    39        if (0 > H5Dwrite_chunk(dsid, H5P_DEFAULT, 0, hchunk_off, cfp.array2d.compressed_size(origarr), cfp.array2d.compressed_data(origarr))) SET_ERROR(H5Dwrite_chunk);
    40
    41        if (0 > H5Dclose(dsid)) SET_ERROR(H5Dclose);
    42
    43        free(buf);
    44        cfp.array2d.dtor(origarr);
    45    }

In particular, look for the line using `H5Dchunk_write` in place of `H5Dwrite`. In all other respects, the code looks the same.

The test case for this code writes uncompressed data as a dataset named `zfparr_original`, the compressed dataset named `zfparr_compressed` using the filter and then the compressed data a second time named `zfparr_direct` using a direct write. Then, the `h5diff` tool is used to compare the data in the original and the direct write datasets.

Note that in order for consumers to work as normal, the producer must set dataset creation properties as it ordinarily would using the H5Z-ZFP filter. In the call to `H5Dchunk_write`, the caller indicates to the HDF5 library not to invoke the filter via the `filters` mask argument.