# ubelt.progiter module

A Progress Iterator

ProgIter lets you measure and print the progress of an iterative process. This can be done either via an iterable interface or using the manual API. Using the iterable interface is most common.

The basic usage of ProgIter is simple and intuitive. Just wrap a python iterable. The following example wraps a `range` iterable and prints reported progress to stdout as the iterable is consumed.

Example

    >>> for n in ProgIter(range(1000)):
    >>>     # do some work
    >>>     pass

Note that by default ProgIter reports information about iteration-rate, fraction-complete, estimated time remaining, time taken so far, and the current wall time.

Example

    >>> # xdoctest: +IGNORE_WANT
    >>> def is_prime(n):
    ...     return n >= 2 and not any(n % i == 0 for i in range(2, n))
    >>> for n in ProgIter(range(1000), verbose=1):
    >>>     # do some work
    >>>     is_prime(n)
    1000/1000... rate=114326.51 Hz, eta=0:00:00, total=0:00:00

For more complex applications is may sometimes be desirable to manually use the ProgIter API. This is done as follows:

Example

    >>> # xdoctest: +IGNORE_WANT
    >>> n = 3
    >>> prog = ProgIter(desc='manual', total=n, verbose=3)
    >>> prog.begin() # Manually begin progress iteration
    >>> for _ in range(n):
    ...     prog.step(inc=1)  # specify the number of steps to increment
    >>> prog.end()  # Manually end progress iteration
    manual 0/3... rate=0 Hz, eta=?, total=0:00:00
    manual 1/3... rate=14454.63 Hz, eta=0:00:00, total=0:00:00
    manual 2/3... rate=17485.42 Hz, eta=0:00:00, total=0:00:00
    manual 3/3... rate=21689.78 Hz, eta=0:00:00, total=0:00:00

When working with ProgIter in either iterable or manual mode you can use the `prog.ensure_newline` method to guarantee that the next call you make to stdout will start on a new line. You can also use the `prog.set_extra` method to update a dynamci “extra” message that is shown in the formatted output. The following example demonstrates this.

Example

    >>> # xdoctest: +IGNORE_WANT
    >>> def is_prime(n):
    ...     return n >= 2 and not any(n % i == 0 for i in range(2, n))
    >>> _iter = range(1000)
    >>> prog = ProgIter(_iter, desc='check primes', verbose=2, show_wall=True)
    >>> for n in prog:
    >>>     if n == 97:
    >>>         print('!!! Special print at n=97 !!!')
    >>>     if is_prime(n):
    >>>         prog.set_extra('Biggest prime so far: {}'.format(n))
    >>>         prog.ensure_newline()
    check primes    0/1000... rate=0 Hz, eta=?, total=0:00:00, wall=2020-10-23 17:27 EST
    check primes    1/1000... rate=95547.49 Hz, eta=0:00:00, total=0:00:00, wall=2020-10-23 17:27 EST
    check primes    4/1000...Biggest prime so far: 3 rate=41062.28 Hz, eta=0:00:00, total=0:00:00, wall=2020-10-23 17:27 EST
    check primes   16/1000...Biggest prime so far: 13 rate=85340.61 Hz, eta=0:00:00, total=0:00:00, wall=2020-10-23 17:27 EST
    check primes   64/1000...Biggest prime so far: 61 rate=164739.98 Hz, eta=0:00:00, total=0:00:00, wall=2020-10-23 17:27 EST
    !!! Special print at n=97 !!!
    check primes  256/1000...Biggest prime so far: 251 rate=206287.91 Hz, eta=0:00:00, total=0:00:00, wall=2020-10-23 17:27 EST
    check primes  512/1000...Biggest prime so far: 509 rate=165271.92 Hz, eta=0:00:00, total=0:00:00, wall=2020-10-23 17:27 EST
    check primes  768/1000...Biggest prime so far: 761 rate=136480.12 Hz, eta=0:00:00, total=0:00:00, wall=2020-10-23 17:27 EST
    check primes 1000/1000...Biggest prime so far: 997 rate=115214.95 Hz, eta=0:00:00, total=0:00:00, wall=2020-10-23 17:27 EST

class ubelt.progiter.ProgIter(iterable=None, desc=None, total=None, freq=1, initial=0, eta_window=64, clearline=True, adjust=True, time_thresh=2.0, show_percent=True, show_times=True, show_rate=True, show_eta=True, show_total=True, show_wall=False, enabled=True, verbose=None, stream=None, chunksize=None, rel_adjust_limit=4.0, homogeneous='auto', timer=None, **kwargs)

  Bases: `_TQDMCompat`, `_BackwardsCompat`

  Prints progress as an iterator progresses

  ProgIter is an alternative to tqdm. ProgIter implements much of the tqdm-API. The main difference between ProgIter and tqdm is that ProgIter does not use threading whereas tqdm does.

  Attributes:

  Note

  Either use ProgIter in a with statement or call prog.end() at the end of the computation if there is a possibility that the entire iterable may not be exhausted.

  Note

  ProgIter is an alternative to tqdm. The main difference between ProgIter and tqdm is that ProgIter does not use threading whereas tqdm does. ProgIter is simpler than tqdm and thus more stable in certain circumstances.

  SeeAlso:

    tqdm - https://pypi.python.org/pypi/tqdm

  References

  [DatagenProgBars]

  http://datagenetics.com/blog/february12017/index.html

  Example

      >>>
      >>> def is_prime(n):
      ...     return n >= 2 and not any(n % i == 0 for i in range(2, n))
      >>> for n in ProgIter(range(100), verbose=1, show_wall=True):
      >>>     # do some work
      >>>     is_prime(n)
      100/100... rate=... Hz, total=..., wall=...

  See attributes more arg information

  Parameters:

    - iterable (List | Iterable) – A list or iterable to loop over

    - desc (str | None) – description label to show with progress

    - total (int | None) – Maximum length of the process. If not specified, we estimate it from the iterable, if possible.

    - freq (int) – How many iterations to wait between messages. Defaults to 1.

    - initial (int) – starting index offset, default=0

    - eta_window (int) – number of previous measurements to use in eta calculation, default=64

    - clearline (bool) – if True messages are printed on the same line otherwise each new progress message is printed on new line. default=True

    - adjust (bool) – if True freq is adjusted based on time_thresh. This may be overwritten depending on the setting of verbose. default=True

    - time_thresh (float) – desired amount of time to wait between messages if adjust is True otherwise does nothing, default=2.0

    - show_percent (bool) – if True show percent progress. Default=True

    - show_times (bool) – if False do not show rate, eta, or wall time. default=True Deprecated. Use show_rate / show_eta / show_wall instead.

    - show_rate (bool) – show / hide rate, default=True

    - show_eta (bool) – show / hide estimated time of arrival (i.e. time to completion), default=True

    - show_wall (bool) – show / hide wall time, default=False

    - stream (typing.IO) – stream where progress information is written to, default=sys.stdout

    - timer (callable) – the timer object to use. Defaults to `time.perf_counter()`.

    - enabled (bool) – if False nothing happens. default=True

    - chunksize (int | None) – indicates that each iteration processes a batch of this size. Iteration rate is displayed in terms of single-items.

    - rel_adjust_limit (float) – Maximum factor update frequency can be adjusted by in a single step. default=4.0

    - verbose (int) – verbosity mode, which controls clearline, adjust, and enabled. The following maps the value of verbose to its effect. 0: enabled=False, 1: enabled=True with clearline=True and adjust=True, 2: enabled=True with clearline=False and adjust=True, 3: enabled=True with clearline=False and adjust=False

    - homogeneous (bool | str) – Indicate if the iterable is likely to take a uniform or homogeneous amount of time per iteration. When True we can enable a speed optimization. When False, the time estimates are more accurate. Default to “auto”, which attempts to determine if it is safe to use True. Has no effect if `adjust` is False.

    - show_total (bool) – if True show total time.

    - **kwargs – accepts most of the tqdm api

  __call__(iterable)

    Overwrites the current iterator with iterable and starts iterating on it.

    Warning

    Using this function is not recommended.

    Parameters:

      iterable (Iterable)

    Returns:

      Iterable

  set_extra(extra)

    specify a custom info appended to the end of the next message

    Parameters:

      extra (str | Callable) – a constant or dynamically constructed extra message.

    Todo

    - [ ] extra is a bad name; come up with something better and rename

    Example

        >>> prog = ProgIter(range(100, 300, 100), show_times=False, verbose=3)
        >>> for n in prog:
        >>>     prog.set_extra('processesing num {}'.format(n))
         0.00% 0/2...
         50.00% 1/2...processesing num 100
         100.00% 2/2...processesing num 200

  _reset_internals()

    Initialize all variables used in the internal state

  begin()

    Initializes information used to measure progress

    This only needs to be used if this ProgIter is not wrapping an iterable. Does nothing if this ProgIter is disabled.

    Returns:

      a chainable self-reference

    Return type:

      ProgIter

  end()

    Signals that iteration has ended and displays the final message.

    This only needs to be used if this ProgIter is not wrapping an iterable. Does nothing if this ProgIter object is disabled or has already finished.

  _iterate()

    iterates with progress

  _homogeneous_check(gen)
  _slow_path_step_body(force=False)
  step(inc=1, force=False)

    Manually step progress update, either directly or by an increment.

    Parameters:

      - inc (int) – number of steps to increment. Defaults to 1.

      - force (bool) – if True forces progress display. Defaults to False.

    Example

        >>> n = 3
        >>> prog = ProgIter(desc='manual', total=n, verbose=3)
        >>> # Need to manually begin and end in this mode
        >>> prog.begin()
        >>> for _ in range(n):
        ...     prog.step()
        >>> prog.end()

    Example

        >>> n = 3
        >>> # can be used as a context manager in manual mode
        >>> with ProgIter(desc='manual', total=n, verbose=3) as prog:
        ...     for _ in range(n):
        ...         prog.step()

  _adjust_frequency()
  _measure_time()

    Measures the current time and update info about how long we’ve been waiting since the last iteration was displayed.

  _update_message_template()
  _build_message_template()

    Defines the template for the progress line

    Returns:

      Tuple[str, str, str]

    Example

        >>> self = ProgIter()
        >>> print(self._build_message_template()[1].strip())
        {desc} {iter_idx:4d}/?...{extra} rate={rate:{rate_format}} Hz, total={total}...

        >>> self = ProgIter(show_total=False, show_eta=False, show_rate=False)
        >>> print(self._build_message_template()[1].strip())
        {desc} {iter_idx:4d}/?...{extra}

        >>> self = ProgIter(total=0, show_times=True)
        >>> print(self._build_message_template()[1].strip())
        {desc} {percent:03.2f}% {iter_idx:1d}/0...{extra} rate={rate:{rate_format}} Hz, total={total}

  format_message()

    Exists only for backwards compatibility.

    See format_message_parts for more recent API.

    Returns:

      str

  format_message_parts()

    builds a formatted progress message with the current values. This contains the special characters needed to clear lines.

    Returns:

      Tuple[str, str, str]

    Example

        >>> self = ProgIter(clearline=False, show_times=False)
        >>> print(repr(self.format_message_parts()[1]))
        '    0/?... '
        >>> self.begin()
        >>> self.step()
        >>> print(repr(self.format_message_parts()[1]))
        ' 1/?... '

    Example

        >>> self = ProgIter(chunksize=10, total=100, clearline=False,
        >>>                 show_times=False, microseconds=True)
        >>> # hack, microseconds=True for coverage, needs real test
        >>> print(repr(self.format_message_parts()[1]))
        ' 0.00% of 10x100... '
        >>> self.begin()
        >>> self.update()  # tqdm alternative to step
        >>> print(repr(self.format_message_parts()[1]))
        ' 1.00% of 10x100... '

  ensure_newline()

    use before any custom printing when using the progress iter to ensure your print statement starts on a new line instead of at the end of a progress line

    Example

        >>> # Unsafe version may write your message on the wrong line
        >>> prog = ProgIter(range(3), show_times=False, freq=2, adjust=False,
        ...                 time_thresh=0)
        >>> for n in prog:
        ...     print('unsafe message')
         0.00% 0/3... unsafe message
        unsafe message
         66.67% 2/3... unsafe message
         100.00% 3/3...
        >>> # apparently the safe version does this too.
        >>> print('---')
        ---
        >>> prog = ProgIter(range(3), show_times=False, freq=2, adjust=False,
        ...                 time_thresh=0)
        >>> for n in prog:
        ...     prog.ensure_newline()
        ...     print('safe message')
         0.00% 0/3...
        safe message
        safe message
         66.67% 2/3...
        safe message
         100.00% 3/3...

  display_message()

    Writes current progress to the output stream

  _tryflush()

    flush to the internal stream

  _write(msg)

    write to the internal stream

    Parameters:

      msg (str) – message to write