# ubelt.util_indexable module

The util_indexable module defines `IndexableWalker` which is a powerful way to iterate through nested Python containers.

RelatedWork:

  - [PypiDictDigger]

References

[PypiDictDigger]

https://pypi.org/project/dict_digger/

[PypiDeepDiff]

https://pypi.org/project/deepdiff/

ubelt.util_indexable._lazy_numpy()

class ubelt.util_indexable.Difference(path: Tuple, value1: Any, value2: Any)

  Bases: `NamedTuple`

  A result class of indexable_diff that organizes what the difference between the indexables is.

  Create new instance of Difference(path, value1, value2)

  path: Tuple

    Alias for field number 0

  value1: Any

    Alias for field number 1

  value2: Any

    Alias for field number 2

  _asdict()

    Return a new dict which maps field names to their values.

  _field_defaults = {}
  _fields = ('path', 'value1', 'value2')
  classmethod _make(iterable)

    Make a new Difference object from a sequence or iterable

  _replace(**kwds)

    Return a new Difference object replacing specified fields with new values

class ubelt.util_indexable.IndexableWalker(data, dict_cls=(<class 'dict'>, ), list_cls=(<class 'list'>, <class 'tuple'>))

  Bases: `Generator`

  Traverses through a nested tree-liked indexable structure.

  Generates a path and value to each node in the structure. The path is a list of indexes which if applied in order will reach the value.

  The `__setitem__` method can be used to modify a nested value based on the path returned by the generator.

  When generating values, you can use “send” to prevent traversal of a particular branch.

  RelatedWork:
    - https://pypi.org/project/python-benedict/ - implements a dictionary

        subclass with similar nested indexing abilities.

  Variables:

    - data (dict | list | tuple) – the wrapped indexable data

    - dict_cls (Tuple[type]) – the types that should be considered dictionary mappings for the purpose of nested iteration. Defaults to `dict`.

    - list_cls (Tuple[type]) – the types that should be considered list-like for the purposes of nested iteration. Defaults to `(list, tuple)`.

    - indexable_cls (Tuple[type]) – combined dict_cls and list_cls

  Example

      >>> import ubelt as ub
      >>> # Given Nested Data
      >>> data = {
      >>>     'foo': {'bar': 1},
      >>>     'baz': [{'biz': 3}, {'buz': [4, 5, 6]}],
      >>> }
      >>> # Create an IndexableWalker
      >>> walker = ub.IndexableWalker(data)
      >>> # We iterate over the data as if it was flat
      >>> # ignore the <want> string due to order issues on older Pythons
      >>> # xdoctest: +IGNORE_WANT
      >>> for path, val in walker:
      >>>     print(path)
      ['foo']
      ['baz']
      ['baz', 0]
      ['baz', 1]
      ['baz', 1, 'buz']
      ['baz', 1, 'buz', 0]
      ['baz', 1, 'buz', 1]
      ['baz', 1, 'buz', 2]
      ['baz', 0, 'biz']
      ['foo', 'bar']
      >>> # We can use "paths" as keys to getitem into the walker
      >>> path = ['baz', 1, 'buz', 2]
      >>> val = walker[path]
      >>> assert val == 6
      >>> # We can use "paths" as keys to setitem into the walker
      >>> assert data['baz'][1]['buz'][2] == 6
      >>> walker[path] = 7
      >>> assert data['baz'][1]['buz'][2] == 7
      >>> # We can use "paths" as keys to delitem into the walker
      >>> assert data['baz'][1]['buz'][1] == 5
      >>> del walker[['baz', 1, 'buz', 1]]
      >>> assert data['baz'][1]['buz'][1] == 7

  Example

      >>> # Create nested data
      >>> # xdoctest: +REQUIRES(module:numpy)
      >>> import numpy as np
      >>> import ubelt as ub
      >>> data = ub.ddict(lambda: int)
      >>> data['foo'] = ub.ddict(lambda: int)
      >>> data['bar'] = np.array([1, 2, 3])
      >>> data['foo']['a'] = 1
      >>> data['foo']['b'] = np.array([1, 2, 3])
      >>> data['foo']['c'] = [1, 2, 3]
      >>> data['baz'] = 3
      >>> print('data = {}'.format(ub.repr2(data, nl=True)))
      >>> # We can walk through every node in the nested tree
      >>> walker = ub.IndexableWalker(data)
      >>> for path, value in walker:
      >>>     print('walk path = {}'.format(ub.repr2(path, nl=0)))
      >>>     if path[-1] == 'c':
      >>>         # Use send to prevent traversing this branch
      >>>         got = walker.send(False)
      >>>         # We can modify the value based on the returned path
      >>>         walker[path] = 'changed the value of c'
      >>> print('data = {}'.format(ub.repr2(data, nl=True)))
      >>> assert data['foo']['c'] == 'changed the value of c'

  Example

      >>> # Test sending false for every data item
      >>> import ubelt as ub
      >>> data = {1: [1, 2, 3], 2: [1, 2, 3]}
      >>> walker = ub.IndexableWalker(data)
      >>> # Sending false means you wont traverse any further on that path
      >>> num_iters_v1 = 0
      >>> for path, value in walker:
      >>>     print('[v1] walk path = {}'.format(ub.repr2(path, nl=0)))
      >>>     walker.send(False)
      >>>     num_iters_v1 += 1
      >>> num_iters_v2 = 0
      >>> for path, value in walker:
      >>>     # When we dont send false we walk all the way down
      >>>     print('[v2] walk path = {}'.format(ub.repr2(path, nl=0)))
      >>>     num_iters_v2 += 1
      >>> assert num_iters_v1 == 2
      >>> assert num_iters_v2 == 8

  Example

      >>> # Test numpy
      >>> # xdoctest: +REQUIRES(CPython)
      >>> # xdoctest: +REQUIRES(module:numpy)
      >>> import ubelt as ub
      >>> import numpy as np
      >>> # By default we don't recurse into ndarrays because they
      >>> # Are registered as an indexable class
      >>> data = {2: np.array([1, 2, 3])}
      >>> walker = ub.IndexableWalker(data)
      >>> num_iters = 0
      >>> for path, value in walker:
      >>>     print('walk path = {}'.format(ub.repr2(path, nl=0)))
      >>>     num_iters += 1
      >>> assert num_iters == 1
      >>> # Currently to use top-level ndarrays, you need to extend what the
      >>> # list class is. This API may change in the future to be easier
      >>> # to work with.
      >>> data = np.random.rand(3, 5)
      >>> walker = ub.IndexableWalker(data, list_cls=(list, tuple, np.ndarray))
      >>> num_iters = 0
      >>> for path, value in walker:
      >>>     print('walk path = {}'.format(ub.repr2(path, nl=0)))
      >>>     num_iters += 1
      >>> assert num_iters == 3 + 3 * 5

  send(arg) → send 'arg' into generator,

    return next yielded value or raise StopIteration.

  throw(typ[, val[, tb]]) → raise exception in generator,

    return next yielded value or raise StopIteration.

    Parameters:

      - typ (Any) – Type of the exception. Should be a `type[BaseException]`, type checking is not working right here.

      - val (Optional[object])

      - tb (Optional[TracebackType])

    Returns:

      Any

    Raises:

      StopIteration –

    References

    [GeneratorThrow]

    https://docs.python.org/3/reference/expressions.html#generator.throw

  keys(non_leaf=False)

    Iterate over the nested paths in the container.

    Parameters:

      non_leaf (bool) – if True, also returns non-leaf indexes, which are indexes to intermediate nested structures. Defaults to False.

    Yields:

      List – a path of indexes into the nested structure

    Example

        >>> import ubelt as ub
        >>> data = {
        >>>     'foo': {'bar': 1},
        >>>     'baz': [{'biz': 3}, 'mid', {'buz': [4, 5, 6]}],
        >>>     'emptynest': [[]],
        >>> }
        >>> keys = list(ub.IndexableWalker(data).keys())
        >>> keys_with_nonleafs = list(ub.IndexableWalker(data).keys(non_leaf=True))
        >>> print(f'keys = {ub.urepr(keys, nl=1)}')
        keys = [
            ['baz', 1],
            ['baz', 2, 'buz', 0],
            ['baz', 2, 'buz', 1],
            ['baz', 2, 'buz', 2],
            ['baz', 0, 'biz'],
            ['foo', 'bar'],
        ]
        >>> assert not any(['emptynest' in p for p in keys])
        >>> assert any(['emptynest' in p for p in keys_with_nonleafs])

  values(non_leaf=False)

    Iterate over the values nested within the container.

    Parameters:

      non_leaf (bool) – if True, also returns non-leaf indexes, which are indexes to intermediate nested structures. Defaults to False.

    Yields:

      Any – the value at each nested path.

    Example

        >>> import ubelt as ub
        >>> data = {
        >>>     'foo': {'bar': 1},
        >>>     'baz': [{'biz': 3}, 'mid', {'buz': [4, 5, 6]}],
        >>>     'emptynest': [[]],
        >>> }
        >>> values = list(ub.IndexableWalker(data).values())
        >>> keys_with_nonleafs = list(ub.IndexableWalker(data).values(non_leaf=True))
        >>> print(values)
        ['mid', 4, 5, 6, 3, 1]
        >>> assert not any(isinstance(v, list) for v in values)
        >>> assert any(isinstance(v, list) for v in keys_with_nonleafs)

  _walk(data=None, prefix=[])

    Defines the underlying generator used by IndexableWalker

    Yields:
      Tuple[List, Any] | None – path (List) - a “path” through the nested data structure

        value (Any) - the value indexed by that “path”.

      Can also yield None in the case that send is called on the generator.

  allclose(other, rel_tol=1e-09, abs_tol=0.0, equal_nan=False, return_info=False)

    Walks through this and another nested data structures and checks if everything is roughly the same.

    Parameters:

      - other (IndexableWalker | List | Dict) – a nested indexable item to compare against.

      - rel_tol (float) – maximum difference for being considered “close”, relative to the magnitude of the input values

      - abs_tol (float) – maximum difference for being considered “close”, regardless of the magnitude of the input values

      - equal_nan (bool) – if True, numpy must be available, and consider nans as equal.

      - return_info (bool) – if True, return extra info dict. Defaults to False.

    Returns:

      A boolean result if `return_info` is false, otherwise a tuple of the boolean result and an “info” dict containing detailed results indicating what matched and what did not.

    Return type:

      bool | Tuple[bool, Dict]

    Example

        >>> import ubelt as ub
        >>> items1 = ub.IndexableWalker({
        >>>     'foo': [1.222222, 1.333],
        >>>     'bar': 1,
        >>>     'baz': [],
        >>> })
        >>> items2 = ub.IndexableWalker({
        >>>     'foo': [1.22222, 1.333],
        >>>     'bar': 1,
        >>>     'baz': [],
        >>> })
        >>> flag, return_info =  items1.allclose(items2, return_info=True)
        >>> print('return_info = {}'.format(ub.repr2(return_info, nl=1)))
        >>> print('flag = {!r}'.format(flag))
        >>> for p1, v1, v2  in return_info['faillist']:
        >>>     v1_ = items1[p1]
        >>>     print('*fail p1, v1, v2 = {}, {}, {}'.format(p1, v1, v2))
        >>> for p1 in return_info['passlist']:
        >>>     v1_ = items1[p1]
        >>>     print('*pass p1, v1_ = {}, {}'.format(p1, v1_))
        >>> assert not flag

        >>> import ubelt as ub
        >>> items1 = ub.IndexableWalker({
        >>>     'foo': [1.0000000000000000000000001, 1.],
        >>>     'bar': 1,
        >>>     'baz': [],
        >>> })
        >>> items2 = ub.IndexableWalker({
        >>>     'foo': [0.9999999999999999, 1.],
        >>>     'bar': 1,
        >>>     'baz': [],
        >>> })
        >>> flag, return_info =  items1.allclose(items2, return_info=True)
        >>> print('return_info = {}'.format(ub.repr2(return_info, nl=1)))
        >>> print('flag = {!r}'.format(flag))
        >>> assert flag

    Example

        >>> import ubelt as ub
        >>> flag, return_info =  ub.IndexableWalker([]).allclose(ub.IndexableWalker([]), return_info=True)
        >>> print('return_info = {!r}'.format(return_info))
        >>> print('flag = {!r}'.format(flag))
        >>> assert flag

    Example

        >>> import ubelt as ub
        >>> flag =  ub.IndexableWalker([]).allclose([], return_info=False)
        >>> print('flag = {!r}'.format(flag))
        >>> assert flag

    Example

        >>> import ubelt as ub
        >>> flag, return_info =  ub.IndexableWalker([]).allclose([1], return_info=True)
        >>> print('return_info = {!r}'.format(return_info))
        >>> print('flag = {!r}'.format(flag))
        >>> assert not flag

    Example

        >>> # xdoctest: +REQUIRES(module:numpy)
        >>> import ubelt as ub
        >>> import numpy as np
        >>> a = np.random.rand(3, 5)
        >>> b = a + 1
        >>> wa = ub.IndexableWalker(a, list_cls=(np.ndarray,))
        >>> wb = ub.IndexableWalker(b, list_cls=(np.ndarray,))
        >>> flag, return_info =  wa.allclose(wb, return_info=True)
        >>> print('return_info = {!r}'.format(return_info))
        >>> print('flag = {!r}'.format(flag))
        >>> assert not flag
        >>> a = np.random.rand(3, 5)
        >>> b = a.copy() + 1e-17
        >>> wa = ub.IndexableWalker([a], list_cls=(np.ndarray, list))
        >>> wb = ub.IndexableWalker([b], list_cls=(np.ndarray, list))
        >>> flag, return_info =  wa.allclose(wb, return_info=True)
        >>> assert flag
        >>> print('return_info = {!r}'.format(return_info))
        >>> print('flag = {!r}'.format(flag))

  diff(other, rel_tol=1e-09, abs_tol=0.0, equal_nan=False)

    Walks through two nested data structures finds differences in the structures.

    Parameters:

      - other (IndexableWalker | List | Dict) – a nested indexable item to compare against.

      - rel_tol (float) – maximum difference for being considered “close”, relative to the magnitude of the input values

      - abs_tol (float) – maximum difference for being considered “close”, regardless of the magnitude of the input values

      - equal_nan (bool) – if True, numpy must be available, and consider nans as equal.

    Returns:

      information about the diff with

        ”similarity”: a score between 0 and 1 “num_differences” being the number of paths not common plus the

        > number of common paths with differing values.

        ”unique1”: being the paths that were unique to self “unique2”: being the paths that were unique to other “faillist”: a list 3-tuples of common path and differing values “num_approximations”:

        > is the number of approximately equal items (i.e. floats) there were

    Return type:

      dict

    Example

        >>> import ubelt as ub
        >>> dct1 = {
        >>>     'foo': [1.222222, 1.333],
        >>>     'bar': 1,
        >>>     'baz': [],
        >>>     'top': [1, 2, 3],
        >>>     'L0': {'L1': {'L2': {'K1': 'V1', 'K2': 'V2', 'D1': 1, 'D2': 2}}},
        >>> }
        >>> dct2 = {
        >>>     'foo': [1.22222, 1.333],
        >>>     'bar': 1,
        >>>     'baz': [],
        >>>     'buz': {1: 2},
        >>>     'top': [1, 1, 2],
        >>>     'L0': {'L1': {'L2': {'K1': 'V1', 'K2': 'V2', 'D1': 10, 'D2': 20}}},
        >>> }
        >>> info = ub.IndexableWalker(dct1).diff(dct2)
        >>> print(f'info = {ub.urepr(info, nl=2)}')

    Example

        >>> # xdoctest: +REQUIRES(module:numpy)
        >>> import ubelt as ub
        >>> import numpy as np
        >>> a = np.random.rand(3, 5)
        >>> b = a + 1
        >>> wa = ub.IndexableWalker(a, list_cls=(np.ndarray,))
        >>> wb = ub.IndexableWalker(b, list_cls=(np.ndarray,))
        >>> info =  wa.diff(wb)
        >>> print(f'info = {ub.urepr(info, nl=2)}')
        >>> a = np.random.rand(3, 5)
        >>> b = a.copy() + 1e-17
        >>> wa = ub.IndexableWalker([a], list_cls=(np.ndarray, list))
        >>> wb = ub.IndexableWalker([b], list_cls=(np.ndarray, list))
        >>> info =  wa.diff(wb)
        >>> print(f'info = {ub.urepr(info, nl=2)}')

    Example

        >>> import ubelt as ub
        >>> # test null similarity
        >>> wa = ub.IndexableWalker({}).diff({})
        >>> assert wa['similarity'] == 1.0

  _abc_impl = <_abc._abc_data object>

ubelt.util_indexable._make_isclose_fn(rel_tol, abs_tol, equal_nan)

ubelt.util_indexable.indexable_allclose(items1, items2, rel_tol=1e-09, abs_tol=0.0, return_info=False)

  Walks through two nested data structures and ensures that everything is roughly the same.

  Note

  Deprecated. Instead use:

    ub.IndexableWalker(items1).allclose(items2)

  Parameters:

    - items1 (dict | list | tuple) – a nested indexable item

    - items2 (dict | list | tuple) – a nested indexable item

    - rel_tol (float) – maximum difference for being considered “close”, relative to the magnitude of the input values

    - abs_tol (float) – maximum difference for being considered “close”, regardless of the magnitude of the input values

    - return_info (bool) – if True, return extra info. Defaults to False.

  Returns:

    A boolean result if `return_info` is false, otherwise a tuple of the boolean result and an “info” dict containing detailed results indicating what matched and what did not.

  Return type:

    bool | Tuple[bool, Dict]

  Example

      >>> import ubelt as ub
      >>> items1 = {
      >>>     'foo': [1.222222, 1.333],
      >>>     'bar': 1,
      >>>     'baz': [],
      >>> }
      >>> items2 = {
      >>>     'foo': [1.22222, 1.333],
      >>>     'bar': 1,
      >>>     'baz': [],
      >>> }
      >>> flag, return_info =  ub.indexable_allclose(items1, items2, return_info=True)
      >>> print('return_info = {}'.format(ub.repr2(return_info, nl=1)))
      >>> print('flag = {!r}'.format(flag))