bm::serializer< BV > Class Template Reference
BitMagic Library » bvector<> container » bvector<> serialization

Bit-vector serialization class. More...

`#include <bmserial.h>`

Inheritance diagram for bm::serializer< BV >:
[legend]

## Data Structures

struct | bookmark_state
| Bookmark state structure. More...

## Public Types

typedef BV | bvector_type
typedef bvector_type::allocator_type | allocator_type
typedef bvector_type::blocks_manager_type | blocks_manager_type
typedef bvector_type::statistics | statistics_type
typedef bvector_type::block_idx_type | block_idx_type
typedef bvector_type::size_type | size_type
typedef byte_buffer< allocator_type > | buffer
typedef bm::bv_ref_vector< BV > | bv_ref_vector_type
typedef bm::xor_sim_model< BV > | xor_sim_model_type
typedef xor_sim_model_type::block_match_chain_type | block_match_chain_type

## Public Member Functions

| serializer (const allocator_type &alloc=allocator_type(), bm::word_t *temp_block=0)
| Constructor.
| serializer (bm::word_t *temp_block)
| ~serializer ()
Compression level settings
void | set_compression_level (unsigned clevel) BMNOEXCEPT
| Set compression level.
unsigned | get_compression_level () const BMNOEXCEPT
| Get current compression level.

## Serialization Methods

size_type | serialize (const BV &bv, unsigned char *buf, size_t buf_size)
| Bitvector serialization into memory block.
void | serialize (const BV &bv, typename serializer< BV >::buffer &buf, const statistics_type *bv_stat=0)
| Bitvector serialization into buffer object (resized automatically).
void | optimize_serialize_destroy (BV &bv, typename serializer< BV >::buffer &buf)
| Bitvector serialization into buffer object (resized automatically) Input bit-vector gets optimized and then destroyed, content is NOT guaranteed after this operation.
const size_type * | get_compression_stat () const BMNOEXCEPT
| Return serialization counter vector.
void | allow_stat_reset (bool allow) BMNOEXCEPT
| Enable/disable statistics reset on each serilaization.
void | reset_compression_stats () BMNOEXCEPT
| Reset all accumulated compression statistics.
void | gap_length_serialization (bool value) BMNOEXCEPT
| Set GAP length serialization (serializes GAP levels of the original vector).
void | byte_order_serialization (bool value) BMNOEXCEPT
| Set byte-order serialization (for cross platform compatibility).
void | set_bookmarks (bool enable, unsigned bm_interval=256) BMNOEXCEPT
| Add skip-markers to serialization BLOB for faster range decode at the expense of some BLOB size increase.
void | set_sparse_cutoff (unsigned cutoff) BMNOEXCEPT
| Fine tuning for Binary Interpolative Compression (levels 5+) The parameter sets average population count per block (64Kbits) below which block is considered very sparse.
void | set_ref_vectors (const bv_ref_vector_type *ref_vect)
| Attach collection of reference vectors for XOR serialization (no transfer of ownership for the pointers).
bool | compute_sim_model (xor_sim_model_type &sim_model, const bv_ref_vector_type &ref_vect, const bm::xor_sim_params &params)
| Calculate XOR similarity model for ref_vector refernece vector must be associated before.
void | set_sim_model (const xor_sim_model_type *sim_model) BMNOEXCEPT
| Atach XOR similarity model (must be computed by the same ref vector).
void | set_curr_ref_idx (size_type ref_idx) BMNOEXCEPT
| Set current index in rer.vector collection (not a row idx or plain idx).
void | encode_header (const BV &bv, bm::encoder &enc) BMNOEXCEPT
| Encode serialization header information.
void | encode_gap_block (const bm::gap_word_t *gap_block, bm::encoder &enc)
void | gamma_gap_block (const bm::gap_word_t *gap_block, bm::encoder &enc) BMNOEXCEPT
void | gamma_gap_array (const bm::gap_word_t *gap_block, unsigned arr_len, bm::encoder &enc, bool inverted=false) BMNOEXCEPT
| Encode GAP block as delta-array with Elias Gamma coder.
void | encode_bit_array (const bm::word_t *block, bm::encoder &enc, bool inverted) BMNOEXCEPT
| Encode bit-block as an array of bits.
void | gamma_gap_bit_block (const bm::word_t *block, bm::encoder &enc) BMNOEXCEPT
void | gamma_arr_bit_block (const bm::word_t *block, bm::encoder &enc, bool inverted) BMNOEXCEPT
void | bienc_arr_bit_block (const bm::word_t *block, bm::encoder &enc, bool inverted) BMNOEXCEPT
void | bienc_arr_sblock (const BV &bv, unsigned sb, bm::encoder &enc) BMNOEXCEPT
void | bienc_gap_bit_block (const bm::word_t *block, bm::encoder &enc) BMNOEXCEPT
| encode bit-block as interpolated bit block of gaps
void | interpolated_arr_bit_block (const bm::word_t *block, bm::encoder &enc, bool inverted) BMNOEXCEPT
void | interpolated_gap_bit_block (const bm::word_t *block, bm::encoder &enc) BMNOEXCEPT
| encode bit-block as interpolated gap block
void | interpolated_gap_array (const bm::gap_word_t *gap_block, unsigned arr_len, bm::encoder &enc, bool inverted) BMNOEXCEPT
| Encode GAP block as an array with binary interpolated coder.
void | interpolated_gap_array_v0 (const bm::gap_word_t *gap_block, unsigned arr_len, bm::encoder &enc, bool inverted) BMNOEXCEPT
void | interpolated_encode_gap_block (const bm::gap_word_t *gap_block, bm::encoder &enc) BMNOEXCEPT
void | encode_bit_interval (const bm::word_t *blk, bm::encoder &enc, unsigned size_control) BMNOEXCEPT
| Encode BIT block with repeatable runs of zeroes.
void | encode_bit_digest (const bm::word_t *blk, bm::encoder &enc, bm::id64_t d0) BMNOEXCEPT
| Encode bit-block using digest (hierarchical compression).
void | encode_xor_match_chain (bm::encoder &enc, const block_match_chain_type &mchain) BMNOEXCEPT
| Encode XOR match chain.
unsigned char | find_gap_best_encoding (const bm::gap_word_t *gap_block) BMNOEXCEPT
| Determine best representation for GAP block based on current set compression level.
unsigned char | find_bit_best_encoding (const bm::word_t *block) BMNOEXCEPT
| Determine best representation for a bit-block.
unsigned char | find_bit_best_encoding_l5 (const bm::word_t *block) BMNOEXCEPT
| Determine best representation for a bit-block (level 5).
void | reset_models () BMNOEXCEPT
void | add_model (unsigned char mod, unsigned score) BMNOEXCEPT
void | xor_tmp_product (const bm::word_t *s_block, const block_match_chain_type &mchain, unsigned i, unsigned j) BMNOEXCEPT
| Compute digest based XOR product, place into tmp XOR block.
static void | process_bookmark (block_idx_type nb, bookmark_state &bookm, bm::encoder &enc) BMNOEXCEPT
| Check if bookmark needs to be placed and if so, encode it into serialization BLOB.

## Detailed Description

template<class BV>
class bm::serializer< BV >

Bit-vector serialization class.

Class designed to convert sparse bit-vectors into a single block of memory ready for file or database storage or network transfer.

Reuse of this class for multiple serializations (but not across threads). Class resue offers some performance advantage (helps with temp memory reallocations).

Examples
  bvsetalgebra.cpp, inv_list.cpp, sample14.cpp, sample22.cpp, sample4.cpp, xsample01.cpp, and xsample07a.cpp.

Definition at line 75 of file bmserial.h.

## Member Typedef Documentation

## allocator_type

template<class BV>

typedef bvector_type::allocator_type bm::serializer< BV >::allocator_type

Definition at line 79 of file bmserial.h.

## block_idx_type

template<class BV>

typedef bvector_type::block_idx_type bm::serializer< BV >::block_idx_type

Definition at line 82 of file bmserial.h.

## block_match_chain_type

template<class BV>

typedef xor_sim_model_type::block_match_chain_type bm::serializer< BV >::block_match_chain_type

Definition at line 90 of file bmserial.h.

## blocks_manager_type

template<class BV>

typedef bvector_type::blocks_manager_type bm::serializer< BV >::blocks_manager_type

Definition at line 80 of file bmserial.h.

## buffer

template<class BV>

typedef byte_buffer<allocator_type> bm::serializer< BV >::buffer

Definition at line 85 of file bmserial.h.

## bv_ref_vector_type

template<class BV>

typedef bm::bv_ref_vector<BV> bm::serializer< BV >::bv_ref_vector_type

Definition at line 86 of file bmserial.h.

## bvector_type

template<class BV>

typedef BV bm::serializer< BV >::bvector_type

Definition at line 78 of file bmserial.h.

## size_type

template<class BV>

typedef bvector_type::size_type bm::serializer< BV >::size_type

Definition at line 83 of file bmserial.h.

## statistics_type

template<class BV>

typedef bvector_type::statistics bm::serializer< BV >::statistics_type

Definition at line 81 of file bmserial.h.

## xor_sim_model_type

template<class BV>

typedef bm::xor_sim_model<BV> bm::serializer< BV >::xor_sim_model_type

Definition at line 87 of file bmserial.h.

## Constructor & Destructor Documentation

## serializer() [1/2]

template<class BV>

bm::serializer< BV >::serializer | ( | const allocator_type & | alloc = allocator_type(),
|  | bm::word_t * | temp_block = 0 )

Constructor.

Parameters

  alloc | - memory allocator
  temp_block | - temporary block for various operations (if NULL it will be allocated and managed by serializer class) Temp block is used as a scratch memory during serialization, use of external temp block allows to avoid unnecessary re-allocations.

Temp block attached is not owned by the class and NOT deallocated on destruction.

Definition at line 1167 of file bmserial.h.

References bm::gap_max_bits, bm::set_compression_default, and bm::sparse_max_l6.

## serializer() [2/2]

template<class BV>

bm::serializer< BV >::serializer | ( | bm::word_t * | temp_block | ) |

Definition at line 1200 of file bmserial.h.

References bm::gap_max_bits, bm::set_compression_default, and bm::sparse_max_l6.

## ~serializer()

template<class BV>

bm::serializer< BV >::~serializer | ( |  | ) |

Definition at line 1232 of file bmserial.h.

## Member Function Documentation

## add_model()

template<class BV>

void bm::serializer< BV >::add_model | ( | unsigned char | mod,
|  | unsigned | score )

protected

Definition at line 1652 of file bmserial.h.

References BM_ASSERT, and BMNOEXCEPT.

Referenced by find_bit_best_encoding(), and find_bit_best_encoding_l5().

## allow_stat_reset()

template<class BV>

void bm::serializer< BV >::allow_stat_reset | ( | bool | allow | ) |

inline

Enable/disable statistics reset on each serilaization.

Definition at line 202 of file bmserial.h.

## bienc_arr_bit_block()

template<class BV>

void bm::serializer< BV >::bienc_arr_bit_block | ( | const bm::word_t * | block,
|  | bm::encoder & | enc,
|  | bool | inverted )

protected

Definition at line 2327 of file bmserial.h.

References bm::bit_block_convert_to_arr(), BMNOEXCEPT, encode_bit_digest(), and interpolated_gap_array().

Referenced by serialize().

## bienc_arr_sblock()

template<class BV>

void bm::serializer< BV >::bienc_arr_sblock | ( | const BV & | bv,
|  | unsigned | sb,
|  | bm::encoder & | enc )

protected

Definition at line 2411 of file bmserial.h.

References bm::bit_out< TEncoder >::bic_encode_u32_cm(), BM_ASSERT, BMNOEXCEPT, bm::convert_sub_to_arr(), bm::sblock_flag_len16, bm::sblock_flag_max16, bm::sblock_flag_max24, bm::sblock_flag_max32, bm::sblock_flag_min16, bm::sblock_flag_min24, bm::sblock_flag_min32, bm::sblock_flag_sb16, bm::sblock_flag_sb32, bm::set_sblock_bienc, and bm::set_sub_total_bits.

Referenced by serialize().

## bienc_gap_bit_block()

template<class BV>

void bm::serializer< BV >::bienc_gap_bit_block | ( | const bm::word_t * | block,
|  | bm::encoder & | enc )

protected

encode bit-block as interpolated bit block of gaps

Definition at line 2352 of file bmserial.h.

References bm::bit_out< TEncoder >::bic_encode_u16(), bm::bit_to_gap(), BM_ASSERT, BMNOEXCEPT, encode_bit_digest(), bm::bit_out< TEncoder >::flush(), bm::gap_max_bits, bm::set_block_bitgap_bienc, and bm::set_block_size.

Referenced by serialize().

## byte_order_serialization()

template<class BV>

void bm::serializer< BV >::byte_order_serialization | ( | bool | value | ) |

Set byte-order serialization (for cross platform compatibility).

Parameters

  value | - TRUE serialization format includes byte-order marker

Examples
  inv_list.cpp, sample4.cpp, and xsample01.cpp.

Definition at line 1278 of file bmserial.h.

References BMNOEXCEPT.

Referenced by compress_inv_dump_file(), convert_bv2bvs(), main(), and bm::serialize().

## compute_sim_model()

template<class BV>

bool bm::serializer< BV >::compute_sim_model | ( | xor_sim_model_type & | sim_model,
|  | const bv_ref_vector_type & | ref_vect,
|  | const bm::xor_sim_params & | params )

Calculate XOR similarity model for ref_vector refernece vector must be associated before.

Parameters

  sim_model | - [out] similarity model to compute
  ref_vect | - [in] reference vectors
  params | - parameters to regulate search depth

Returns
  true - if similarity model created successfully

See also
  set_ref_vectors

Definition at line 1313 of file bmserial.h.

## encode_bit_array()

template<class BV>

void bm::serializer< BV >::encode_bit_array | ( | const bm::word_t * | block,
|  | bm::encoder & | enc,
|  | bool | inverted )

protected

Encode bit-block as an array of bits.

Definition at line 2283 of file bmserial.h.

References bm::bit_block_convert_to_arr(), BMNOEXCEPT, encode_bit_digest(), bm::set_block_arrbit, and bm::set_block_arrbit_inv.

Referenced by serialize().

## encode_bit_digest()

template<class BV>

void bm::serializer< BV >::encode_bit_digest | ( | const bm::word_t * | blk,
|  | bm::encoder & | enc,
|  | bm::id64_t | d0 )

protected

Encode bit-block using digest (hierarchical compression).

Definition at line 2108 of file bmserial.h.

References bm::bmi_blsi_u64(), bm::bmi_bslr_u64(), BMNOEXCEPT, encode_bit_interval(), bm::set_block_bit, bm::set_block_bit_digest0, bm::set_block_digest_wave_size, bm::set_block_size, and bm::word_bitcount64().

Referenced by bienc_arr_bit_block(), bienc_gap_bit_block(), encode_bit_array(), interpolated_arr_bit_block(), and serialize().

## encode_bit_interval()

template<class BV>

void bm::serializer< BV >::encode_bit_interval | ( | const bm::word_t * | blk,
|  | bm::encoder & | enc,
|  | unsigned | size_control )

protected

Encode BIT block with repeatable runs of zeroes.

Definition at line 2056 of file bmserial.h.

References BM_ASSERT, BMNOEXCEPT, bm::set_block_bit_0runs, and bm::set_block_size.

Referenced by encode_bit_digest(), and serialize().

## encode_gap_block()

template<class BV>

void bm::serializer< BV >::encode_gap_block | ( | const bm::gap_word_t * | gap_block,
|  | bm::encoder & | enc )

protected

Encode GAP block

Definition at line 1996 of file bmserial.h.

References BM_ASSERT, BM_FALLTHROUGH, find_gap_best_encoding(), gamma_gap_array(), gamma_gap_block(), bm::gap_convert_to_arr(), bm::gap_equiv_len, interpolated_encode_gap_block(), interpolated_gap_array(), bm::encoder::put_16(), bm::encoder::put_8(), bm::set_block_arrgap, bm::set_block_arrgap_bienc, bm::set_block_arrgap_bienc_inv, bm::set_block_arrgap_egamma, bm::set_block_arrgap_egamma_inv, bm::set_block_arrgap_inv, bm::set_block_bit_1bit, bm::set_block_gap, and bm::set_block_gap_bienc.

Referenced by serialize().

## encode_header()

template<class BV>

void bm::serializer< BV >::encode_header | ( | const BV & | bv,
|  | bm::encoder & | enc )

protected

Encode serialization header information.

Definition at line 1333 of file bmserial.h.

References bm::BM_HM_64_BIT, bm::BM_HM_DEFAULT, bm::BM_HM_HXOR, bm::BM_HM_NO_BO, bm::BM_HM_NO_GAPL, bm::BM_HM_RESIZE, BMNOEXCEPT, bm::globals< T >::byte_order(), bm::gap_levels, and bm::id_max.

Referenced by serialize().

## encode_xor_match_chain()

template<class BV>

void bm::serializer< BV >::encode_xor_match_chain | ( | bm::encoder & | enc,
|  | const block_match_chain_type & | mchain )

protected

Encode XOR match chain.

Definition at line 2161 of file bmserial.h.

References BM_ASSERT, BMNOEXCEPT, bm::check_pair_vect_vbr(), and bm::set_block_xor_chain.

Referenced by serialize().

## find_bit_best_encoding()

template<class BV>

unsigned char bm::serializer< BV >::find_bit_best_encoding | ( | const bm::word_t * | block | ) |

protected

Determine best representation for a bit-block.

Definition at line 1777 of file bmserial.h.

References add_model(), bm::best_representation(), bm::bit_block_change_bc(), bm::bit_block_count(), bm::bit_count_nonzero_size(), bm::block_waves, BM_ASSERT, BMNOEXCEPT, bm::calc_block_digest0(), bm::compute_s_block_descr(), bm::e_bit_0, bm::e_bit_1, bm::e_bit_bit, bm::e_bit_end, bm::e_bit_GAP, bm::e_bit_IINT, bm::e_bit_INT, find_bit_best_encoding_l5(), bm::gap_equiv_len, bm::gap_max_bits, bm::gap_max_buff_len, reset_models(), bm::block_waves_xor_descr::sb_bc, bm::block_waves_xor_descr::sb_gc, bm::set_block_aone, bm::set_block_arrbit, bm::set_block_arrbit_inv, bm::set_block_arrgap_egamma, bm::set_block_arrgap_egamma_inv, bm::set_block_azero, bm::set_block_bit, bm::set_block_bit_0runs, bm::set_block_bit_1bit, bm::set_block_bit_digest0, bm::set_block_digest_wave_size, bm::set_block_gap_egamma, bm::set_block_size, and bm::word_bitcount64().

Referenced by serialize().

## find_bit_best_encoding_l5()

template<class BV>

unsigned char bm::serializer< BV >::find_bit_best_encoding_l5 | ( | const bm::word_t * | block | ) |

protected

Determine best representation for a bit-block (level 5).

Definition at line 1661 of file bmserial.h.

References add_model(), bm::bit_block_change_bc(), bm::bit_count_nonzero_size(), BMNOEXCEPT, bm::calc_block_digest0(), bm::gap_max_bits, bm::gap_max_buff_len, bm::set_block_aone, bm::set_block_arr_bienc, bm::set_block_arr_bienc_inv, bm::set_block_arrbit, bm::set_block_arrbit_inv, bm::set_block_arrgap, bm::set_block_arrgap_bienc, bm::set_block_arrgap_bienc_inv, bm::set_block_arrgap_egamma, bm::set_block_arrgap_egamma_inv, bm::set_block_arrgap_inv, bm::set_block_azero, bm::set_block_bit, bm::set_block_bit_0runs, bm::set_block_bit_1bit, bm::set_block_bit_digest0, bm::set_block_bitgap_bienc, bm::set_block_gap_bienc, bm::set_block_gap_egamma, bm::set_block_size, and bm::word_bitcount64().

Referenced by find_bit_best_encoding().

## find_gap_best_encoding()

template<class BV>

unsigned char bm::serializer< BV >::find_gap_best_encoding | ( | const bm::gap_word_t * | gap_block | ) |

protected

Determine best representation for GAP block based on current set compression level.

Returns
  set_block_bit, set_block_bit_1bit, set_block_arrgap set_block_arrgap_egamma, set_block_arrgap_bienc set_block_arrgap_inv, set_block_arrgap_egamma_inv set_block_arrgap_bienc_inv, set_block_gap_egamma set_block_gap_bienc

Definition at line 1945 of file bmserial.h.

References BMNOEXCEPT, bm::gap_bit_count_unr(), bm::gap_length(), bm::gap_max_bits, bm::set_block_arrgap, bm::set_block_arrgap_bienc, bm::set_block_arrgap_bienc_inv, bm::set_block_arrgap_egamma, bm::set_block_arrgap_egamma_inv, bm::set_block_arrgap_inv, bm::set_block_bit_1bit, bm::set_block_gap, bm::set_block_gap_bienc, and bm::set_block_gap_egamma.

Referenced by encode_gap_block().

## gamma_arr_bit_block()

template<class BV>

void bm::serializer< BV >::gamma_arr_bit_block | ( | const bm::word_t * | block,
|  | bm::encoder & | enc,
|  | bool | inverted )

protected

Definition at line 2311 of file bmserial.h.

References bm::bit_block_convert_to_arr(), BMNOEXCEPT, gamma_gap_array(), bm::set_block_bit, and bm::set_block_size.

Referenced by serialize().

## gamma_gap_array()

template<class BV>

void bm::serializer< BV >::gamma_gap_array | ( | const bm::gap_word_t * | gap_block,
|  | unsigned | arr_len,
|  | bm::encoder & | enc,
|  | bool | inverted = false )

protected

Encode GAP block as delta-array with Elias Gamma coder.

Definition at line 1484 of file bmserial.h.

References BMNOEXCEPT, bm::bit_out< TEncoder >::gamma(), bm::set_block_arrgap, bm::set_block_arrgap_egamma, bm::set_block_arrgap_egamma_inv, and bm::set_block_arrgap_inv.

Referenced by encode_gap_block(), and gamma_arr_bit_block().

## gamma_gap_bit_block()

template<class BV>

void bm::serializer< BV >::gamma_gap_bit_block | ( | const bm::word_t * | block,
|  | bm::encoder & | enc )

protected

Definition at line 2302 of file bmserial.h.

References bm::bit_to_gap(), BM_ASSERT, BMNOEXCEPT, gamma_gap_block(), and bm::gap_equiv_len.

Referenced by serialize().

## gamma_gap_block()

template<class BV>

void bm::serializer< BV >::gamma_gap_block | ( | const bm::gap_word_t * | gap_block,
|  | bm::encoder & | enc )

protected

Encode GAP block with Elias Gamma coder

Definition at line 1445 of file bmserial.h.

References BMNOEXCEPT, bm::for_each_dgap(), bm::gap_length(), bm::set_block_gap, and bm::set_block_gap_egamma.

Referenced by encode_gap_block(), and gamma_gap_bit_block().

## gap_length_serialization()

template<class BV>

void bm::serializer< BV >::gap_length_serialization | ( | bool | value | ) |

Set GAP length serialization (serializes GAP levels of the original vector).

Parameters

  value | - when TRUE serialized vector includes GAP levels parameters

Examples
  inv_list.cpp, sample4.cpp, and xsample01.cpp.

Definition at line 1272 of file bmserial.h.

References BMNOEXCEPT.

Referenced by compress_inv_dump_file(), convert_bv2bvs(), main(), bm::compressed_collection_serializer< CBC >::serialize(), and bm::serialize().

## get_compression_level()

template<class BV>

unsigned bm::serializer< BV >::get_compression_level | ( |  | ) | const

inline

Get current compression level.

Definition at line 133 of file bmserial.h.

## get_compression_stat()

template<class BV>

const size_type * bm::serializer< BV >::get_compression_stat | ( |  | ) | const

inline

Return serialization counter vector.

Definition at line 196 of file bmserial.h.

## interpolated_arr_bit_block()

template<class BV>

void bm::serializer< BV >::interpolated_arr_bit_block | ( | const bm::word_t * | block,
|  | bm::encoder & | enc,
|  | bool | inverted )

protected

Definition at line 2502 of file bmserial.h.

References bm::bit_out< TEncoder >::bic_encode_u16(), bm::bit_block_convert_to_arr(), BM_ASSERT, BMNOEXCEPT, encode_bit_digest(), bm::bit_out< TEncoder >::flush(), bm::set_block_arr_bienc, bm::set_block_arr_bienc_8bh, bm::set_block_arr_bienc_inv, and bm::set_block_size.

Referenced by serialize().

## interpolated_encode_gap_block()

template<class BV>

void bm::serializer< BV >::interpolated_encode_gap_block | ( | const bm::gap_word_t * | gap_block,
|  | bm::encoder & | enc )

protected

Encode GAP block with using binary interpolated encoder

Definition at line 1385 of file bmserial.h.

References bm::bit_out< TEncoder >::bic_encode_u16(), BM_ASSERT, BMNOEXCEPT, bm::bit_out< TEncoder >::flush(), bm::gap_length(), bm::set_block_gap, bm::set_block_gap_bienc, and bm::set_block_gap_bienc_v2.

Referenced by encode_gap_block(), and interpolated_gap_bit_block().

## interpolated_gap_array()

template<class BV>

void bm::serializer< BV >::interpolated_gap_array | ( | const bm::gap_word_t * | gap_block,
|  | unsigned | arr_len,
|  | bm::encoder & | enc,
|  | bool | inverted )

protected

Encode GAP block as an array with binary interpolated coder.

Definition at line 1577 of file bmserial.h.

References bm::bit_out< TEncoder >::bic_encode_u16(), BM_ASSERT, BMNOEXCEPT, bm::bit_out< TEncoder >::flush(), interpolated_gap_array_v0(), bm::set_block_arrgap, bm::set_block_arrgap_bienc_inv_v2, bm::set_block_arrgap_bienc_v2, and bm::set_block_arrgap_inv.

Referenced by bienc_arr_bit_block(), and encode_gap_block().

## interpolated_gap_array_v0()

template<class BV>

void bm::serializer< BV >::interpolated_gap_array_v0 | ( | const bm::gap_word_t * | gap_block,
|  | unsigned | arr_len,
|  | bm::encoder & | enc,
|  | bool | inverted )

protected

Definition at line 1529 of file bmserial.h.

References bm::bit_out< TEncoder >::bic_encode_u16(), BM_ASSERT, BMNOEXCEPT, bm::bit_out< TEncoder >::flush(), bm::bit_out< TEncoder >::gamma(), bm::set_block_arrgap, bm::set_block_arrgap_bienc, bm::set_block_arrgap_bienc_inv, and bm::set_block_arrgap_inv.

Referenced by interpolated_gap_array().

## interpolated_gap_bit_block()

template<class BV>

void bm::serializer< BV >::interpolated_gap_bit_block | ( | const bm::word_t * | block,
|  | bm::encoder & | enc )

protected

encode bit-block as interpolated gap block

Definition at line 2342 of file bmserial.h.

References bm::bit_to_gap(), BM_ASSERT, BMNOEXCEPT, bm::gap_max_bits, and interpolated_encode_gap_block().

Referenced by serialize().

## optimize_serialize_destroy()

template<class BV>

void bm::serializer< BV >::optimize_serialize_destroy | ( | BV & | bv,
|  | typename serializer< BV >::buffer & | buf )

Bitvector serialization into buffer object (resized automatically) Input bit-vector gets optimized and then destroyed, content is NOT guaranteed after this operation.

Effectively it moves data into the buffer.

The reason this operation exsists is because it is faster to do all three operations in one single pass. This is a destructive serialization!

Parameters

  bv | - input/output bitvector
  buf | - output buffer object

Examples
  sample4.cpp.

Definition at line 2265 of file bmserial.h.

References bm::serialize().

Referenced by main().

## process_bookmark()

template<class BV>

void bm::serializer< BV >::process_bookmark | ( | block_idx_type | nb,
|  | bookmark_state & | bookm,
|  | bm::encoder & | enc )

static protected

Check if bookmark needs to be placed and if so, encode it into serialization BLOB.

Parameters

  nb | - block idx
  bookm | - bookmark state structure
  enc | - BLOB encoder

Definition at line 2593 of file bmserial.h.

References BM_ASSERT, BMNOEXCEPT, bm::set_nb_bookmark16, bm::set_nb_bookmark24, bm::set_nb_bookmark32, bm::set_nb_sync_mark16, bm::set_nb_sync_mark24, bm::set_nb_sync_mark32, bm::set_nb_sync_mark48, bm::set_nb_sync_mark64, and bm::set_nb_sync_mark8.

Referenced by serialize().

## reset_compression_stats()

template<class BV>

void bm::serializer< BV >::reset_compression_stats | ( |  | ) |

Reset all accumulated compression statistics.

Definition at line 1244 of file bmserial.h.

References BMNOEXCEPT.

Referenced by serialize().

## reset_models()

template<class BV>

void bm::serializer< BV >::reset_models | ( |  | ) |

inline protected

Definition at line 383 of file bmserial.h.

Referenced by find_bit_best_encoding().

## serialize() [1/2]

template<class BV>

void bm::serializer< BV >::serialize | ( | const BV & | bv,
|  | typename serializer< BV >::buffer & | buf,
|  | const statistics_type * | bv_stat = 0 )

Bitvector serialization into buffer object (resized automatically).

Parameters

  bv | - input bitvector
  buf | - output buffer object
  bv_stat | - input (optional) bit-vector statistics object if NULL, serialize will compute the statistics

Definition at line 2241 of file bmserial.h.

References BM_ASSERT, bm::bv_statistics::max_serialize_mem, and bm::serialize().

## serialize() [2/2]

template<class BV>

serializer< BV >::size_type bm::serializer< BV >::serialize | ( | const BV & | bv,
|  | unsigned char * | buf,
|  | size_t | buf_size )

Bitvector serialization into memory block.

Parameters

  bv | - input bitvector
  buf | - out buffer (pre-allocated) No range checking is done in this method. It is responsibility of caller to allocate sufficient amount of memory using information from calc_stat() function.
  buf_size | - size of the output buffer

Returns
  Size of serialization block.

See also
  calc_stat

Examples
  bvsetalgebra.cpp, sample14.cpp, sample22.cpp, sample4.cpp, xsample01.cpp, and xsample07a.cpp.

Definition at line 2703 of file bmserial.h.

References bienc_arr_bit_block(), bienc_arr_sblock(), bienc_gap_bit_block(), bm::bit_block_find(), BM_ASSERT, bm::BM_HM_SPARSE, BM_IS_GAP, BM_SER_NEXT_GRP, BMGAP_PTR, bm::block_match_chain< BLOCK_IDX >::chain_size, bm::check_block_one(), bm::check_block_zero(), bm::e_no_xor_match, bm::e_xor_match_BC, bm::e_xor_match_EQ, bm::e_xor_match_GC, bm::e_xor_match_iBC, encode_bit_array(), encode_bit_digest(), encode_bit_interval(), encode_gap_block(), encode_header(), encode_xor_match_chain(), find_bit_best_encoding(), FULL_BLOCK_FAKE_ADDR, gamma_arr_bit_block(), gamma_gap_bit_block(), bm::get_block_coord(), interpolated_arr_bit_block(), interpolated_gap_bit_block(), bm::block_match_chain< BLOCK_IDX >::match, bm::block_match_chain< BLOCK_IDX >::nb, process_bookmark(), bm::encoder::put_16(), bm::encoder::put_32(), bm::encoder::put_64(), bm::encoder::put_8(), bm::encoder::put_8_16_32(), bm::encoder::put_prefixed_array_32(), bm::block_match_chain< BLOCK_IDX >::ref_idx, reset_compression_stats(), bm::set_block_16one, bm::set_block_16zero, bm::set_block_1one, bm::set_block_1zero, bm::set_block_32one, bm::set_block_32zero, bm::set_block_64one, bm::set_block_64zero, bm::set_block_8one, bm::set_block_8zero, bm::set_block_aone, bm::set_block_arr_bienc, bm::set_block_arr_bienc_inv, bm::set_block_arrbit, bm::set_block_arrbit_inv, bm::set_block_arrgap_bienc, bm::set_block_arrgap_bienc_inv, bm::set_block_arrgap_egamma, bm::set_block_arrgap_egamma_inv, bm::set_block_azero, bm::set_block_bit, bm::set_block_bit_0runs, bm::set_block_bit_1bit, bm::set_block_bit_digest0, bm::set_block_bitgap_bienc, bm::set_block_end, bm::set_block_gap_bienc, bm::set_block_gap_egamma, bm::set_block_ref_eq, bm::set_block_size, bm::set_block_xor_ref16, bm::set_block_xor_ref16_um, bm::set_block_xor_ref32, bm::set_block_xor_ref32_um, bm::set_block_xor_ref8, bm::set_block_xor_ref8_um, bm::encoder::set_pos(), bm::set_sub_array_size, bm::set_total_blocks, bm::encoder::size(), bm::block_match_chain< BLOCK_IDX >::xor_d64, and xor_tmp_product().

Referenced by convert_bv2bvs(), generate_k_mers(), main(), make_BLOB(), bm::compressed_collection_serializer< CBC >::serialize(), and bm::serialize().

## set_bookmarks()

template<class BV>

void bm::serializer< BV >::set_bookmarks | ( | bool | enable,
|  | unsigned | bm_interval = 256 )

Add skip-markers to serialization BLOB for faster range decode at the expense of some BLOB size increase.

Parameters

  enable | - TRUE searilization will add bookmark codes
  bm_interval | - bookmark interval in (number of blocks) suggested values between 4 and 512 (block size is 64K bits) smaller interval means more bookmarks added to the skip list allows faster range deserialization at the expense of somewhat increased BLOB size.

Examples
  sample22.cpp, and xsample07a.cpp.

Definition at line 1284 of file bmserial.h.

References BMNOEXCEPT.

Referenced by generate_k_mers(), and main().

## set_compression_level()

template<class BV>

void bm::serializer< BV >::set_compression_level | ( | unsigned | clevel | ) |

Set compression level.

Higher compression takes more time to process.

Parameters

  clevel | - compression level (0-6) 0 - take as is 1, 2 - apply light weight RLE/GAP encodings, limited depth hierarchical compression, intervals encoding 3 - variant of 2 with different cut-offs 4 - delta transforms plus Elias Gamma encoding where possible legacy) 5 - Binary Interpolative Coding (BIC) - light settings 6 - Binary Interpolative Coding (BIC) - harder settings

See also
  get_compression_level

Examples
  inv_list.cpp, and xsample01.cpp.

Definition at line 1251 of file bmserial.h.

References BMNOEXCEPT, bm::set_compression_max, bm::sparse_max_l5, and bm::sparse_max_l6.

Referenced by compress_inv_dump_file(), convert_bv2bvs(), main(), and make_BLOB().

## set_curr_ref_idx()

template<class BV>

void bm::serializer< BV >::set_curr_ref_idx | ( | size_type | ref_idx | ) |

Set current index in rer.vector collection (not a row idx or plain idx).

Definition at line 1327 of file bmserial.h.

References BMNOEXCEPT.

## set_ref_vectors()

template<class BV>

void bm::serializer< BV >::set_ref_vectors | ( | const bv_ref_vector_type * | ref_vect | ) |

Attach collection of reference vectors for XOR serialization (no transfer of ownership for the pointers).

Definition at line 1299 of file bmserial.h.

References bm::set_block_size.

## set_sim_model()

template<class BV>

void bm::serializer< BV >::set_sim_model | ( | const xor_sim_model_type * | sim_model | ) |

Atach XOR similarity model (must be computed by the same ref vector).

Definition at line 1321 of file bmserial.h.

References BMNOEXCEPT.

## set_sparse_cutoff()

template<class BV>

void bm::serializer< BV >::set_sparse_cutoff | ( | unsigned | cutoff | ) |

Fine tuning for Binary Interpolative Compression (levels 5+) The parameter sets average population count per block (64Kbits) below which block is considered very sparse.

If super block (group of 256 blocks) is very sparse it applies block size expansion (for the compression purposes) to improve compression rates.

Definition at line 1262 of file bmserial.h.

References BM_ASSERT, BMNOEXCEPT, and bm::sparse_max_l6.

## xor_tmp_product()

template<class BV>

void bm::serializer< BV >::xor_tmp_product | ( | const bm::word_t * | s_block,
|  | const block_match_chain_type & | mchain,
|  | unsigned | i,
|  | unsigned | j )

protected

Compute digest based XOR product, place into tmp XOR block.

Definition at line 2206 of file bmserial.h.

References bm::bit_block_xor(), BM_IS_GAP, BMGAP_PTR, BMNOEXCEPT, and bm::gap_convert_to_bitset().

Referenced by serialize().

The documentation for this class was generated from the following file:

- bmserial.h