# ubelt.util_cmd module

This module exposes the `ubelt.cmd()` command, which provides a simple means for interacting with the command line. This uses `subprocess.Popen` under the hood, but improves upon existing `subprocess` functionality by:

(1) Adding the option to “tee” the output, i.e. simultaneously capture and write to stdout and stderr.

(2) Always specify the command as a string. The `subprocess` module expects the command as either a `List[str]` if `shell=False` and `str` if `shell=True`. If necessary, `ubelt.util_cmd.cmd()` will automatically convert from one format to the other, so passing in either case will work.

(3) Specify if the process blocks or not by setting `detach`. Note: when `detach is True` it is not possible to tee the output.

Example

    >>> import ubelt as ub
    >>> # Running with verbose=1 will write to stdout in real time
    >>> info = ub.cmd('echo "write your command naturally"', verbose=1)
    write your command naturally
    >>> # The return type is a dictionary of information depending
    >>> # on how `ub.cmd` was invoked.
    >>> print('info = ' + ub.repr2(info))
    info = {
        'command': 'echo "write your command naturally"',
        'cwd': None,
        'err': '',
        'out': 'write your command naturally\n',
        'proc': <...Popen...>,
        'ret': 0,
    }

The cmd is able to handle common uses cases of the subprocess module with a simpler interface.

    import subprocess
    import ubelt as ub

Run without capturing output and without printing to the screen

    # stdlib
    subprocess.run(['ls', '-l'], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, universal_newlines=True)

    # ubelt equivalent
    ub.cmd(['ls', '-l'], capture=False)

Print output to the screen, but no programmatic access to the data

    # stdlib
    subprocess.check_call(['ls', '-l'])

    # ubelt equivalent
    ub.cmd(['ls', '-l'], verbose=1, capture=False)

Get programmatic access to the data but don’t show it on screen

    # stdlib
    subprocess.check_output(['ls', '-l'], universal_newlines=True)

    # ubelt equivalent
    ub.cmd(['ls', '-l'])['out']

Get programmatic access AND show it on screen

    # stdlib has no easy way to to this

    # ubelt has "tee" functionality
    ub.cmd(['ls', '-l'], verbose=1)

ubelt.util_cmd.cmd(command, shell=False, detach=False, verbose=0, tee=None, cwd=None, env=None, tee_backend='auto', check=False, system=False, timeout=None, capture=True)

  Executes a command in a subprocess.

  The advantage of this wrapper around subprocess is that (1) you control if the subprocess prints to stdout, (2) the text written to stdout and stderr is returned for parsing, (3) cross platform behavior that lets you specify the command as a string or tuple regardless of whether or not shell=True. (4) ability to detach, return the process object and allow the process to run in the background (eventually we may return a Future object instead).

  Parameters:

    - command (str | List[str]) – command string, tuple of executable and args, or shell command.

    - shell (bool) – if True, process is run in shell. Defaults to False. This is required if your command is shell-specific (e.g. uses bash pipe), and in which case the backend shell depends on the operating system.

    - detach (bool) – if True, process is detached and run in background. Defaults to False.

    - verbose (int) – verbosity mode. Can be 0, 1, 2, or 3. Defaults to 0.

    - tee (bool | None) – if True, simultaneously writes to stdout while capturing output from the command. If not specified, defaults to True if verbose > 0. If detach is True, then this argument is ignored.

    - cwd (str | PathLike | None) – Path to run command. Defaults to current working directory if unspecified.

    - env (Dict[str, str] | None) – environment passed to Popen

    - tee_backend (str) – backend for tee output. Valid choices are: “auto”, “select” (POSIX only), and “thread”. Defaults to “auto”.

    - check (bool) – if True, check that the return code was zero before returning, otherwise raise a `subprocess.CalledProcessError`. Does nothing if detach is True. Defaults to False.

    - system (bool) – if True, most other considerations are dropped, and `os.system()` is used to execute the command in a platform dependent way. Other arguments such as env, tee, timeout, and shell are all ignored. Defaults to False. (New in version 1.1.0)

    - timeout (float | None) – If the process does not complete in `timeout` seconds, raise a `subprocess.TimeoutExpired`. (New in version 1.1.0).

    - capture (bool) – if True, the stdout/stderr are captured and returned in the information dictionary. Ignored if detach or system is True.

  Returns:

    info - information about command status. if detach is False `info` contains captured standard out, standard error, and the return code if detach is True `info` contains a reference to the process.

  Return type:

    dict | CmdOutput

  Raises:

    - ValueError - on an invalid configuration –

    - subprocess.TimeoutExpired - if the timeout limit is exceeded –

    - subprocess.CalledProcessError - if check and the return value is non zero –

  Note

  When using the tee output, the stdout and stderr may be shuffled from what they would be on the command line.

  Note

  While this function is generally compatible with subprocess.run and other variants of Popen, we force defaults of universal_newlines=True, and choose the values of stdout and stderr based on other arguments. We are considering the pros and cons of a completely drop-in-replacement API.

  Related Work:

    Similar to other libraries: [SubprocTee], [ShellJob], [CmdRunner], [PyInvoke].

  References

  [SO_11495783]

  https://stackoverflow.com/questions/11495783/redirect-subprocess-stderr-to-stdout

  [SO_7729336]

  https://stackoverflow.com/questions/7729336/how-can-i-print-and-display-subprocess-stdout-and-stderr-output-without-distorti

  [SO_33560364]

  https://stackoverflow.com/questions/33560364/python-windows-parsing-command-lines-with-shlex

  [SubprocTee]

  https://github.com/pycontribs/subprocess-tee

  [ShellJob]

  https://github.com/mortoray/shelljob

  [CmdRunner]

  https://github.com/netinvent/command_runner

  [PyInvoke]

  https://www.pyinvoke.org/prior-art.html

  CommandLine

      xdoctest -m ubelt.util_cmd cmd:6
      python -c "import ubelt as ub; ub.cmd('ping localhost -c 2', verbose=2)"
      pytest "$(python -c 'import ubelt; print(ubelt.util_cmd.__file__)')" -sv --xdoctest-verbose 2

  Example

      >>> import ubelt as ub
      >>> info = ub.cmd(('echo', 'simple cmdline interface'), verbose=1)
      simple cmdline interface
      >>> assert info['ret'] == 0
      >>> assert info['out'].strip() == 'simple cmdline interface'
      >>> assert info['err'].strip() == ''

  Example

      >>> import ubelt as ub
      >>> info = ub.cmd('echo str noshell', verbose=0)
      >>> assert info['out'].strip() == 'str noshell'

  Example

      >>> # windows echo will output extra single quotes
      >>> import ubelt as ub
      >>> info = ub.cmd(('echo', 'tuple noshell'), verbose=0)
      >>> assert info['out'].strip().strip("'") == 'tuple noshell'

  Example

      >>> # Note this command is formatted to work on win32 and unix
      >>> import ubelt as ub
      >>> info = ub.cmd('echo str&&echo shell', verbose=0, shell=True)
      >>> assert info['out'].strip() == 'str' + chr(10) + 'shell'

  Example

      >>> import ubelt as ub
      >>> info = ub.cmd(('echo', 'tuple shell'), verbose=0, shell=True)
      >>> assert info['out'].strip().strip("'") == 'tuple shell'

  Example

      >>> import pytest
      >>> import ubelt as ub
      >>> info = ub.cmd('echo hi', check=True)
      >>> import subprocess
      >>> with pytest.raises(subprocess.CalledProcessError):
      >>>     ub.cmd('exit 1', check=True, shell=True)

  Example

      >>> import ubelt as ub
      >>> from os.path import join, exists
      >>> dpath = ub.Path.appdir('ubelt', 'test').ensuredir()
      >>> fpath1 = (dpath / 'cmdout1.txt').delete()
      >>> fpath2 = (dpath / 'cmdout2.txt').delete()
      >>> # Start up two processes that run simultaneously in the background
      >>> info1 = ub.cmd(('touch', str(fpath1)), detach=True)
      >>> info2 = ub.cmd('echo writing2 > ' + str(fpath2), shell=True, detach=True)
      >>> # Detached processes are running in the background
      >>> # We can run other code while we wait for them.
      >>> while not exists(fpath1):
      ...     pass
      >>> while not exists(fpath2):
      ...     pass
      >>> # communicate with the process before you finish
      >>> # (otherwise you may leak a text wrapper)
      >>> info1['proc'].communicate()
      >>> info2['proc'].communicate()
      >>> # Check that the process actually did finish
      >>> assert (info1['proc'].wait()) == 0
      >>> assert (info2['proc'].wait()) == 0
      >>> # Check that the process did what we expect
      >>> assert fpath1.read_text() == ''
      >>> assert fpath2.read_text().strip() == 'writing2'

  Example

      >>> # Can also use ub.cmd to call os.system
      >>> import pytest
      >>> import ubelt as ub
      >>> import subprocess
      >>> info = ub.cmd('echo hi', check=True, system=True)
      >>> with pytest.raises(subprocess.CalledProcessError):
      >>>     ub.cmd('exit 1', check=True, shell=True)