Skip to content
Back to top
View this page

# Persistent Hashing and Persistent Dictionaries

This module contains functionality that allows hashing with keys that remain valid across interpreter invocations, unlike Python’s built-in hashes.

This module also provides a disk-backed dictionary that uses persistent hashing.

exception pytools.persistent_dict.NoSuchEntryError

  Raised when an entry is not found in a `PersistentDict`.

exception pytools.persistent_dict.NoSuchEntryCollisionError

  Raised when an entry is not found in a `PersistentDict`, but it contains an entry with the same hash key (hash collision).

exception pytools.persistent_dict.ReadOnlyEntryError

  Raised when an attempt is made to overwrite an entry in a `WriteOncePersistentDict`.

exception pytools.persistent_dict.CollisionWarning

  Warning raised when a collision is detected in a `PersistentDict`.

exception pytools.persistent_dict.NanKeyWarning

  Warning raised when a NaN is encountered while hashing a key in `KeyBuilder`.

class pytools.persistent_dict.KeyBuilder

  A (stateless) object that computes persistent hashes of objects fed to it. Subclassing this class permits customizing the computation of hash keys.

  This class follows the same general rules as Python’s built-in hashing:

  - Only immutable objects can be hashed.

  - If two objects compare equal, they must hash to the same value.

  - Objects with the same hash may or may not compare equal.

  In addition, hashes computed with `KeyBuilder` have the following properties:

  - The hash is persistent across interpreter invocations.

  - The hash is the same across different Python versions and platforms.

  - The hash is invariant with respect to `PYTHONHASHSEED`.

  - Hashes are computed using functionality from `hashlib`.

  Key builders of this type are used by `PersistentDict`, but other uses are entirely allowable.

  __call__(key: Any) → str

    Return the hash of key.

  rec(key_hash: Hash, key: Any) → Hash
    Parameters:

      - key_hash – the hash object to be updated with the hash of key.

      - key – the (immutable) Python object to be hashed.

    Returns:

      the updated key_hash

    Changed in version 2021.2: Now returns the updated key_hash.

  static new_hash()

    Return a new hash instance following the protocol of the ones from `hashlib`. This will permit switching to different hash algorithms in the future. Subclasses are expected to use this to create new hashes. Not doing so is deprecated and may stop working as early as 2022.

    Added in version 2021.2.

class pytools.persistent_dict.PersistentDict(identifier: str, key_builder: KeyBuilder | None = None, container_dir: str | None = None, *, enable_wal: bool = False, safe_sync: bool | None = None)

  A concurrent disk-backed dictionary.

  Note

  This class intentionally does not store all values with a certain key, based on the assumption that key conflicts are highly unlikely, and if they occur, almost always due to a bug in the hash key generation code (`KeyBuilder`).

  Warning

  Since NaNs compare unequal to themselves, keys that include NaNs can not be retrieved from a `PersistentDict` and will lead to a `NoSuchEntryCollisionError` on retrieval.

  __init__(identifier: str, key_builder: KeyBuilder | None = None, container_dir: str | None = None, *, enable_wal: bool = False, safe_sync: bool | None = None) → None
    Parameters:

      - identifier – a filename-compatible string identifying this dictionary

      - key_builder – a subclass of `KeyBuilder`

      - container_dir – the directory in which to store this dictionary. If `None`, the default cache directory from `platformdirs.user_cache_dir()` is used

      - enable_wal – enable write-ahead logging (WAL) mode. This mode is faster than the default rollback journal mode, but it is not compatible with network filesystems.

  __getitem__(key: K) → V

    Return the value associated with key in the dictionary.

  __setitem__(key: K, value: V) → None

    Store (key, value) in the dictionary.

  __delitem__(key: K) → None

    Remove the entry associated with key from the dictionary.

  clear() → None

    Remove all entries from the dictionary.

  store(key: K, value: V, _skip_if_present: bool = False) → None

    Store (key, value) in the dictionary.

  store_if_not_present(key: K, value: V) → None

    Store (key, value) if key is not already present.

  fetch(key: K) → V

    Return the value associated with key in the dictionary.

  remove(key: K) → None

    Remove the entry associated with key from the dictionary.

class pytools.persistent_dict.WriteOncePersistentDict(identifier: str, key_builder: KeyBuilder | None = None, container_dir: str | None = None, *, enable_wal: bool = False, safe_sync: bool | None = None, in_mem_cache_size: int = 256)

  A concurrent disk-backed dictionary that disallows overwriting/ deletion (but allows removing all entries).

  Compared with `PersistentDict`, this class has faster retrieval times because it uses an LRU cache to cache entries in memory.

  Note

  This class intentionally does not store all values with a certain key, based on the assumption that key conflicts are highly unlikely, and if they occur, almost always due to a bug in the hash key generation code (`KeyBuilder`).

  Warning

  Since NaNs compare unequal to themselves, keys that include NaNs can not be retrieved from a `WriteOncePersistentDict` and will lead to a `NoSuchEntryCollisionError` on retrieval.

  __init__(identifier: str, key_builder: KeyBuilder | None = None, container_dir: str | None = None, *, enable_wal: bool = False, safe_sync: bool | None = None, in_mem_cache_size: int = 256) → None
    Parameters:

      - identifier – a filename-compatible string identifying this dictionary

      - key_builder – a subclass of `KeyBuilder`

      - container_dir – the directory in which to store this dictionary. If `None`, the default cache directory from `platformdirs.user_cache_dir()` is used

      - enable_wal – enable write-ahead logging (WAL) mode. This mode is faster than the default rollback journal mode, but it is not compatible with network filesystems.

      - in_mem_cache_size – retain an in-memory cache of up to in_mem_cache_size items (with an LRU replacement policy)

  __getitem__(key: K) → V

    Return the value associated with key in the dictionary.

  __setitem__(key: K, value: V) → None

    Store (key, value) in the dictionary.

  clear() → None

    Remove all entries from the dictionary.

  clear_in_mem_cache() → None

    Clear the in-memory cache of this dictionary.

    Added in version 2023.1.1.

  store(key: K, value: V, _skip_if_present: bool = False) → None

    Store (key, value) in the dictionary.

  store_if_not_present(key: K, value: V) → None

    Store (key, value) if key is not already present.

  fetch(key: K) → V

    Return the value associated with key in the dictionary.

## Internal stuff that is only here because the documentation tool wants it

class pytools.persistent_dict.K

  A type variable for the key type of a `PersistentDict`.

class pytools.persistent_dict.V

  A type variable for the value type of a `PersistentDict`.