Hide navigation sidebar
Hide table of contents sidebar
Back to top
Toggle table of contents sidebar

# A Collection of Utilities

## Math

pytools.levi_civita(tup: Tuple[int, ...]) → int

  Compute an entry of the Levi-Civita symbol for the indices tuple.

pytools.perm(*args, **kwargs)

pytools.comb(*args, **kwargs)

## Assertive accessors

pytools.one(iterable: Iterable[T]) → T

  Return the first entry of iterable. Assert that iterable has only that one entry.

pytools.is_single_valued(iterable: ~typing.Iterable[~pytools.T], equality_pred: ~typing.Callable[[~pytools.T, ~pytools.T], bool] = <built-in function eq>) → bool

pytools.all_roughly_equal(iterable, threshold)

pytools.single_valued(iterable: ~typing.Iterable[~pytools.T], equality_pred: ~typing.Callable[[~pytools.T, ~pytools.T], bool] = <built-in function eq>) → T

  Return the first entry of iterable; Assert that other entries are the same with the first entry of iterable.

## Memoization

pytools.memoize(*args: F, **kwargs: Any) → F

  Stores previously computed function values in a cache.

  Two keyword-only arguments are supported:

  Parameters:

    - use_kwargs – Allows the caller to use keyword arguments. Defaults to `False`. Setting this to `True` has a non-negligible performance impact.

    - key – A function receiving the same arguments as the decorated function which computes and returns the cache key.

pytools.memoize_on_first_arg(function: Callable[[Concatenate[T, P]], R], *, cache_dict_name: str | None = None) → Callable[[Concatenate[T, P]], R]

  Like `memoize_method()`, but for functions that take the object in which do memoization information is stored as first argument.

  Supports cache deletion via `function_name.clear_cache(self)`.

pytools.memoize_method(method: Callable[[Concatenate[T, P]], R]) → Callable[[Concatenate[T, P]], R]

  Supports cache deletion via `method_name.clear_cache(self)`.

  Changed in version 2021.2: Can memoize methods on classes that do not allow setting attributes (e.g. by overwritting `__setattr__`), e.g. frozen `dataclasses`.

pytools.memoize_in(container: Any, identifier: Hashable) → None

  Adds a cache to the function it decorates. The cache is attached to container and must be uniquely specified by identifier (i.e. all functions using the same container and identifier will be using the same cache). The decorated function may only receive positional arguments.

  Note

  This function works well on nested functions, which do not have stable global identifiers.

  Changed in version 2020.3: identifier no longer needs to be a `str`, but it needs to be hashable.

  Changed in version 2021.2.1: Can now use instances of classes as container that do not allow setting attributes (e.g. by overwritting `__setattr__`), e.g. frozen `dataclasses`.

pytools.keyed_memoize_on_first_arg(key: Callable[[P], Hashable], *, cache_dict_name: str | None = None) → None

  Like `memoize_method()`, but for functions that take the object in which memoization information is stored as first argument.

  Supports cache deletion via `function_name.clear_cache(self)`.

  Parameters:

    - key – A function receiving the same arguments as the decorated function which computes and returns the cache key.

    - cache_dict_name – The name of the dict attribute in the instance used to hold the cache.

  New in version 2020.3.

pytools.keyed_memoize_method(key: Callable[[P], Hashable], *, cache_dict_name: str | None = None) → None

  Like `memoize_method`, but additionally uses a function key to compute the key under which the function result is stored.

  Supports cache deletion via `method_name.clear_cache(self)`.

  Parameters:

    key – A function receiving the same arguments as the decorated function which computes and returns the cache key.

  New in version 2020.3.

  Changed in version 2021.2: Can memoize methods on classes that do not allow setting attributes (e.g. by overwritting `__setattr__`), e.g. frozen `dataclasses`.

pytools.keyed_memoize_in(container: Any, identifier: Hashable, key: Callable[[P], Hashable]) → None

  Like `memoize_in`, but additionally uses a function key to compute the key under which the function result is memoized.

  Parameters:

    key – A function receiving the same arguments as the decorated function which computes and returns the cache key.

  New in version 2021.2.1.

## Argmin/max

pytools.argmin2(iterable, return_value=False)

pytools.argmax2(iterable, return_value=False)

pytools.argmin(iterable)

pytools.argmax(iterable)

## Cartesian products

pytools.cartesian_product(*args)

pytools.distinct_pairs(list1, list2)

## Permutations, Tuples, Integer sequences

pytools.wandering_element(length, wanderer=1, landscape=0)

pytools.generate_nonnegative_integer_tuples_below(n, length=None, least=0)

  n may be a sequence, in which case length must be None.

pytools.generate_nonnegative_integer_tuples_summing_to_at_most(n, length)

  Enumerate all non-negative integer tuples summing to at most n, exhausting the search space by varying the first entry fastest, and the last entry the slowest.

pytools.generate_all_integer_tuples_below(n, length, least_abs=0)

pytools.generate_permutations(original)

  Generate all permutations of the list original.

  Nicked from http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/252178

pytools.generate_unique_permutations(original)

  Generate all unique permutations of the list original.

## Formatting

class pytools.Table(alignments: Tuple[str, ...] | None = None)

  An ASCII table generator.

  nrows
  ncolumns
  alignments

    A `tuple` of alignments of each column: `"l"`, `"c"`, or `"r"`, for left, center, and right alignment, respectively). Columns which have no alignment specifier will use the last specified alignment. For example, with `alignments=("l", "r")`, the third and all following columns will use right alignment.

  add_row(row: Tuple[Any, ...]) → None
  __str__() → str

    Returns a string representation of the table.

        >>> tbl = Table(alignments=['l', 'r', 'l'])
        >>> tbl.add_row([1, '|'])
        >>> tbl.add_row([10, '20||'])
        >>> print(tbl)
        1  |    |
        ---+------
        10 | 20||

  github_markdown() → str

    Returns a string representation of the table formatted as GitHub-Flavored Markdown.

        >>> tbl = Table(alignments=['l', 'r', 'l'])
        >>> tbl.add_row([1, '|'])
        >>> tbl.add_row([10, '20||'])
        >>> print(tbl.github_markdown())
        1  |     \|
        :--|-------:
        10 | 20\|\|

  csv(dialect: str = 'excel', csv_kwargs: Dict[str, Any] | None = None) → str

    Returns a string containing a CSV representation of the table.

    Parameters:

      - dialect – String passed to `csv.writer()`.

      - csv_kwargs – Dict of arguments passed to `csv.writer()`.

        >>> tbl = Table()
        >>> tbl.add_row([1, ","])
        >>> tbl.add_row([10, 20])
        >>> print(tbl.csv())
        1,","
        10,20

  latex(skip_lines: int = 0, hline_after: Tuple[int, ...] | None = None) → str

    Returns a string containing the rows of a LaTeX representation of the table.

    Parameters:

      - skip_lines – number of lines to skip at the start of the table.

      - hline_after – list of row indices after which to add an `hline` (the indices must subtract skip_lines, if non-zero).

        >>> tbl = Table()
        >>> tbl.add_row([0, "skipped"])
        >>> tbl.add_row([1, "apple"])
        >>> tbl.add_row([2, "pear"])
        >>> print(tbl.latex(skip_lines=1))
        1 & apple \\
        2 & pear \\

pytools.merge_tables(*tables: Table, skip_columns: Tuple[int, ...] | None = None) → Table
  Parameters:

    skip_columns – a `tuple` of column indices to skip in all the tables except the first one.

pytools.string_histogram(iterable, min_value=None, max_value=None, bin_count=20, width=70, bin_starts=None, use_unicode=True)

pytools.word_wrap(text, width, wrap_using='\n')

  A word-wrap function that preserves existing line breaks and most spaces in the text. Expects that existing line breaks are posix newlines (`\n`).

## Debugging

pytools.typedump(val: Any, max_seq: int = 5, special_handlers: Mapping[Type, Callable] | None = None, fully_qualified_name: bool = True) → str

  Return a string representation of the type of val, recursing into iterable objects.

  Parameters:

    - val – The object for which the type should be returned.

    - max_seq – For iterable objects, the maximum number of elements to include in the return string. Lower this value if you get a `RecursionError`.

    - special_handlers – An optional mapping of specific types to special handlers.

    - fully_qualified_name – Return fully qualified names, that is, include module names and use `__qualname__` instead of `__name__`.

  Returns:

    A string representation of the type of val.

pytools.invoke_editor(s, filename='edit.txt', descr='the file')

## Progress bars

class pytools.ProgressBar(descr, total, initial=0, length=40)
  draw()
  progress(steps=1)
  set_progress(done)
  finished()
  __enter__()
  __exit__(exc_type, exc_val, exc_tb)

## Name generation

pytools.generate_unique_names(prefix)

pytools.generate_numbered_unique_names(prefix: str, num: int | None = None) → Iterable[Tuple[int, str]]

class pytools.UniqueNameGenerator(existing_names: Set[str] | None = None, forced_prefix: str = '')

  Class that creates a new `str` on each `__call__()` that is unique to the generator.

  __init__(existing_names: Set[str] | None = None, forced_prefix: str = '')

    Create a new `UniqueNameGenerator`.

    Parameters:

      - existing_names – a `set` of existing names that will be skipped when generating new names.

      - forced_prefix – all generated `str` have this prefix.

  is_name_conflicting(name: str) → bool

    Returns True if name conflicts with an existing `str`.

  add_name(name: str, *, conflicting_ok: bool = False) → None
    Parameters:

      conflicting_ok – A flag to dictate the behavior when name is conflicting with the set of existing names. If True, a conflict is silently passed. If False, a `ValueError` is raised on encountering a conflict.

  add_names(names: Iterable[str], *, conflicting_ok: bool = False) → None
    Parameters:

      conflicting_ok – Plainly passed to `UniqueNameGenerator.add_name()`.

  __call__(based_on: str = 'id') → str

    Returns a new unique name.

## Deprecation Warnings

pytools.deprecate_keyword(oldkey: str, newkey: str | None = None, *, deadline: str | None = None)

  Decorator used to deprecate function keyword arguments.

  Parameters:

    - oldkey – deprecated argument name.

    - newkey – new argument name that serves the same purpose, if any.

    - deadline – expected time frame for the removal of the deprecated argument.

## Functions for dealing with (large) auxiliary files

pytools.download_from_web_if_not_present(url, local_name=None)

  New in version 2017.5.

## Helpers for `numpy`

pytools.reshaped_view(a, newshape)

  Create a new view object with shape `newshape` without copying the data of `a`. This function is different from `numpy.reshape` by raising an exception when data copy is necessary.

  Parameters:

    - a – a `numpy.ndarray` object.

    - newshape – an `int` object or a tuple of `int` objects.

  New in version 2018.4.

## Timing data

class pytools.ProcessTimer

  Measures elapsed wall time and process time.

  __enter__()
  __exit__(exc_type, exc_val, exc_tb)
  done()

  Timing data attributes:

  wall_elapsed
  process_elapsed

  New in version 2018.5.

## Log utilities

class pytools.ProcessLogger(logger, description, silent_level=None, noisy_level=None, long_threshold_seconds=None)

  Logs the completion time of a (presumably) lengthy process to `logging`. Only uses a high log level if the process took perceptible time.

  __init__(logger, description, silent_level=None, noisy_level=None, long_threshold_seconds=None)
  done(extra_msg=None, *extra_fmt_args)
  __enter__()
  __exit__(exc_type, exc_val, exc_tb)

class pytools.DebugProcessLogger(logger, description, silent_level=None, noisy_level=None, long_threshold_seconds=None)

class pytools.log_process(logger, description=None, long_threshold_seconds=None)

  A decorator that uses `ProcessLogger` to log data about calls to the wrapped function.

  __init__(logger, description=None, long_threshold_seconds=None)
  __call__(wrapped)

    Call self as a function.

## Sorting in natural order

pytools.natorder(item)

  Return a key for natural order string comparison.

  See `natsorted()`.

  New in version 2020.1.

pytools.natsorted(iterable, key=None, reverse=False)

  Sort using natural order [1], as opposed to lexicographic order.

  Example:

      >>> sorted(["_10", "_1", "_9"]) == ["_1", "_10", "_9"]
      True
      >>> natsorted(["_10", "_1", "_9"]) == ["_1", "_9", "_10"]
      True

  Parameters:

    - iterable – an iterable to be sorted. It must only have strings, unless key is specified.

    - key – if provided, a key function that returns strings for ordering using natural order.

    - reverse – if True, sorts in descending order.

  Returns:

    a sorted list

  [1]

  https://en.wikipedia.org/wiki/Natural_sort_order

  New in version 2020.1.

## Backports of newer Python functionality

pytools.resolve_name(name)

  A backport of `pkgutil.resolve_name()` (added in Python 3.9).

  New in version 2021.1.2.

## Hashing

pytools.unordered_hash(hash_instance, iterable, hash_constructor=None)

  Using a hash algorithm given by the parameter-less constructor hash_constructor, return a hash object whose internal state depends on the entries of iterable, but not their order. If hash is the instance returned by evaluating `hash_constructor()`, then the each entry i of the iterable must permit `hash.upate(i)` to succeed. An example of hash_constructor is `hashlib.sha256` from `hashlib`. `hash.digest_size` must also be defined. If hash_constructor is not provided, `hash_instance.name` is used to deduce it.

  Returns:

    the updated hash_instance.

  Warning

  The construction used in this function is likely not cryptographically secure. Do not use this function in a security-relevant context.

  New in version 2021.2.

## Sampling

pytools.sphere_sample_equidistant(npoints_approx: int, r: float = 1.0)

  Generate points regularly distributed on a sphere based on https://www.cmu.edu/biolphys/deserno/pdf/sphere_equi.pdf.

  Returns:

    an `ndarray` of shape `(3, npoints)`, where `npoints` does not generally equal npoints_approx.

pytools.sphere_sample_fibonacci(npoints: int, r: float = 1.0, *, optimize: str | None = None)

  Generate points on a sphere based on an offset Fibonacci lattice from [2].

  [2]

  http://extremelearning.com.au/how-to-evenly-distribute-points-on-a-sphere-more-effectively-than-the-canonical-fibonacci-lattice/

  Parameters:

    optimize – takes the values: None to use the standard Fibonacci lattice, `"minimum"` to minimize the nearest neighbor distances in the lattice and `"average"` to minimize the average distances in the lattice.

  Returns:

    an `ndarray` of shape `(3, npoints)`.

## String utilities

pytools.strtobool(val: str | None, default: bool | None = None) → bool

  Convert a string representation of truth to True or False. True values are ‘y’, ‘yes’, ‘t’, ‘true’, ‘on’, and ‘1’; false values are ‘n’, ‘no’, ‘f’, ‘false’, ‘off’, and ‘0’. Uppercase versions are also accepted. If default is None, raises ValueError if val is anything else. If val is None and default is not None, returns default. Based on `distutils.util.strtobool()`.

  Parameters:

    - val – Value to convert.

    - default – Value to return if val is None.

  Returns:

    Truth value of val.

## Type Variables Used

class pytools.T

class pytools.R

  Generic unbound invariant `typing.TypeVar`.

class pytools.F

  Generic invariant `typing.TypeVar` bound to a `typing.Callable`.

class pytools.P

  Generic unbound invariant `typing.ParamSpec`.

# An in-memory relational database table

class pytools.datatable.DataTable(column_names: Sequence[str], column_data: List[Any] | None = None)

  An in-memory relational database table.

  __init__(column_names: Sequence[str], column_data: List[Any] | None = None) → None

    Construct a new table, with the given C{column_names}.

    Parameters:

      - column_names – An indexable of column name strings.

      - column_data – None or a list of tuples of the same length as column_names indicating an initial set of data.

  copy() → DataTable

    Make a copy of the instance, but leave individual rows untouched.

    If the rows are modified later, they will also be modified in the copy.

  deep_copy() → DataTable

    Make a copy of the instance down to the row level.

    The copy’s rows may be modified independently from the original.

  join(column: str, other_column: str, other_table: DataTable, outer: bool = False) → DataTable

    Return a table joining this and the C{other_table} on C{column}.

    The new table has the following columns: - C{column}, titled the same as in this table. - the columns of this table, minus C{column}. - the columns of C{other_table}, minus C{other_column}.

    Assumes both tables are sorted ascendingly by the column by which they are joined.

# Dot helper functions

pytools.graphviz.dot_escape(s: str) → str

  Escape the string s for compatibility with the dot language, particularly backslashes and HTML tags.

  Parameters:

    s – The input string to escape.

  Returns:

    s with special characters escaped.

pytools.graphviz.show_dot(dot_code: str, output_to: str | None = None) → str | None

  Visualize the graph represented by dot_code.

  Parameters:

    - dot_code – An instance of `str` in the dot language to visualize.

    - output_to –

      An instance of `str` that can be one of:

      - `"xwindow"` to visualize the graph as an X window.

      - `"browser"` to visualize the graph as an SVG file in the system’s default web-browser.

      - `"svg"` to store the dot code as an SVG file on the file system. Returns the path to the generated SVG file.

      Defaults to `"xwindow"` if X11 support is present, otherwise defaults to `"browser"`.

  Returns:

    Depends on output_to. If `"svg"`, returns the path to the generated SVG file, otherwise returns `None`.