Skip to content

# cmd2.parsing

## `` cmd2.parsing

Statement parsing classes for cmd2.

### `` MacroArg `dataclass`

    MacroArg(start_index, number_str, is_escaped)

Information used to resolve or unescape macro arguments.

#### `` start_index `instance-attribute`

    start_index

#### `` number_str `instance-attribute`

    number_str

#### `` is_escaped `instance-attribute`

    is_escaped

#### `` macro_normal_arg_pattern `class-attribute`

    macro_normal_arg_pattern = compile('(?<!{){\\d+}|{\\d+}(?!})')

#### `` macro_escaped_arg_pattern `class-attribute`

    macro_escaped_arg_pattern = compile('{{2}\\d+}{2}')

#### `` digit_pattern `class-attribute`

    digit_pattern = compile('\\d+')

### `` Macro `dataclass`

    Macro(name, value, minimum_arg_count, args=tuple())

Defines a cmd2 macro.

#### `` name `instance-attribute`

    name

#### `` value `instance-attribute`

    value

#### `` minimum_arg_count `instance-attribute`

    minimum_arg_count

#### `` args `class-attribute` `instance-attribute`

    args = field(default_factory=tuple)

### `` Statement `dataclass`

    Statement(args='', raw='', command='', multiline_command=False, terminator='', suffix='', redirector='', redirect_to='')

Bases: `str`

String subclass with additional attributes to store the results of parsing.

Instances of this class should not be created by anything other than the StatementParser.parse method, nor should any of the attributes be modified once the object is created.

The string portion of the class contains the arguments, but not the command, nor the output redirection clauses.

Tips:

1. `argparse <https://docs.python.org/3/library/argparse.html>`_ is your friend for anything complex. `cmd2` has the decorator (cmd2.decorators.with_argparser) which you can use to make your command method receive a namespace of parsed arguments, whether positional or denoted with switches.

2. For commands with simple positional arguments, use args or arg_list

3. If you don't want to have to worry about quoted arguments, see argv for a trick which strips quotes off for you.

#### `` args `class-attribute` `instance-attribute`

    args = ''

#### `` raw `class-attribute` `instance-attribute`

    raw = ''

#### `` command `class-attribute` `instance-attribute`

    command = ''

#### `` multiline_command `class-attribute` `instance-attribute`

    multiline_command = False

#### `` terminator `class-attribute` `instance-attribute`

    terminator = ''

#### `` suffix `class-attribute` `instance-attribute`

    suffix = ''

#### `` redirector `class-attribute` `instance-attribute`

    redirector = ''

#### `` redirect_to `class-attribute` `instance-attribute`

    redirect_to = ''

#### `` command_and_args `property`

    command_and_args

Combine command and args with a space separating them.

Quoted arguments remain quoted. Output redirection and piping are excluded, as are any command terminators.

#### `` post_command `property`

    post_command

A string containing any ending terminator, suffix, and redirection chars.

#### `` expanded_command_line `property`

    expanded_command_line

Concatenate cmd2.parsing.Statement.command_and_args and cmd2.parsing.Statement.post_command.

#### `` argv `property`

    argv

A list of arguments a-la `sys.argv`.

The first element of the list is the command after shortcut and macro expansion. Subsequent elements of the list contain any additional arguments, with quotes removed, just like bash would. This is very useful if you are going to use `argparse.parse_args()`.

If you want to strip quotes from the input, you can use `argv[1:]`.

#### `` arg_list `property`

    arg_list

Return the arguments in a list (quotes preserved).

#### `` to_dict

    to_dict()

Convert this Statement into a dictionary for use in persistent JSON history files.

Source code in `cmd2/parsing.py`

    220
    221
    222

    def to_dict(self) -> dict[str, Any]:
        """Convert this Statement into a dictionary for use in persistent JSON history files."""
        return asdict(self)

#### `` from_dict `classmethod`

    from_dict(source_dict)

Restore a Statement from a dictionary.

PARAMETER | DESCRIPTION
--- | ---
`source_dict` | source data dictionary (generated using to_dict()) TYPE: `dict[str, Any]`

RETURNS | DESCRIPTION
--- | ---
`Self` | Statement object

Source code in `cmd2/parsing.py`

    224
    225
    226
    227
    228
    229
    230
    231
    232
    233
    234
    235
    236
    237
    238
    239

    @classmethod
    def from_dict(cls, source_dict: dict[str, Any]) -> Self:
        """Restore a Statement from a dictionary.

        :param source_dict: source data dictionary (generated using to_dict())
        :return: Statement object
        """
        # value needs to be passed as a positional argument. It corresponds to the args field.
        try:
            value = source_dict["args"]
        except KeyError:
            raise KeyError("Statement dictionary is missing 'args' field") from None

        # Filter out 'args' so it isn't passed twice
        kwargs = {k: v for k, v in source_dict.items() if k != "args"}
        return cls(value, **kwargs)

### `` PartialStatement `dataclass`

    PartialStatement(command, args, raw, multiline_command)

A partially parsed command line.

This separates the command from its arguments without validating terminators, redirection, or quoted string completion.

Note: Unlike cmd2.parsing.Statement, this is a simple data object and does not inherit from [str][].

#### `` command `instance-attribute`

    command

#### `` args `instance-attribute`

    args

#### `` raw `instance-attribute`

    raw

#### `` multiline_command `instance-attribute`

    multiline_command

#### `` command_and_args `property`

    command_and_args

Combine command and args with a space between them.

### `` StatementParser

    StatementParser(terminators=None, multiline_commands=None, aliases=None, shortcuts=None)

Parse user input as a string into discrete command components.

Initialize an instance of StatementParser.

The following will get converted to an immutable tuple before storing internally: terminators, multiline commands, and shortcuts.

PARAMETER | DESCRIPTION
--- | ---
`terminators` | iterable containing strings which should terminate commands TYPE: `Iterable[str] | None` DEFAULT: `None`
`multiline_commands` | iterable containing the names of commands that accept multiline input TYPE: `Iterable[str] | None` DEFAULT: `None`
`aliases` | dictionary containing aliases TYPE: `Mapping[str, str] | None` DEFAULT: `None`
`shortcuts` | dictionary containing shortcuts TYPE: `Mapping[str, str] | None` DEFAULT: `None`

Source code in `cmd2/parsing.py`

    279
    280
    281
    282
    283
    284
    285
    286
    287
    288
    289
    290
    291
    292
    293
    294
    295
    296
    297
    298
    299
    300
    301
    302
    303
    304
    305
    306
    307
    308
    309
    310
    311
    312
    313
    314
    315
    316
    317
    318
    319
    320
    321
    322
    323
    324
    325
    326
    327
    328
    329
    330
    331
    332
    333
    334
    335
    336
    337
    338
    339
    340
    341
    342
    343

    def __init__(
        self,
        terminators: Iterable[str] | None = None,
        multiline_commands: Iterable[str] | None = None,
        aliases: Mapping[str, str] | None = None,
        shortcuts: Mapping[str, str] | None = None,
    ) -> None:
        """Initialize an instance of StatementParser.

        The following will get converted to an immutable tuple before storing internally:
        terminators, multiline commands, and shortcuts.

        :param terminators: iterable containing strings which should terminate commands
        :param multiline_commands: iterable containing the names of commands that accept multiline input
        :param aliases: dictionary containing aliases
        :param shortcuts: dictionary containing shortcuts
        """
        self.terminators: tuple[str, ...]
        if terminators is None:
            self.terminators = (constants.MULTILINE_TERMINATOR,)
        else:
            self.terminators = tuple(terminators)
        self.multiline_commands: tuple[str, ...] = tuple(multiline_commands) if multiline_commands is not None else ()
        self.aliases: dict[str, str] = dict(aliases) if aliases is not None else {}

        if shortcuts is None:
            shortcuts = constants.DEFAULT_SHORTCUTS

        # Sort the shortcuts in descending order by name length because the longest match
        # should take precedence. (e.g., @@file should match '@@' and not '@'.
        self.shortcuts = tuple(sorted(shortcuts.items(), key=lambda x: len(x[0]), reverse=True))

        # commands have to be a word, so make a regular expression
        # that matches the first word in the line. This regex has three
        # parts:
        #     - the '\A\s*' matches the beginning of the string (even
        #       if contains multiple lines) and gobbles up any leading
        #       whitespace
        #     - the first parenthesis enclosed group matches one
        #       or more non-whitespace characters with a non-greedy match
        #       (that's what the '+?' part does). The non-greedy match
        #       ensures that this first group doesn't include anything
        #       matched by the second group
        #     - the second parenthesis group must be dynamically created
        #       because it needs to match either whitespace, something in
        #       REDIRECTION_CHARS, one of the terminators, or the end of
        #       the string (\Z matches the end of the string even if it
        #       contains multiple lines)
        #
        invalid_command_chars = (
            *constants.QUOTES,
            *constants.REDIRECTION_CHARS,
            *self.terminators,
        )

        # escape each item so it will for sure get treated as a literal
        second_group_items = [re.escape(x) for x in invalid_command_chars]
        # add the whitespace and end of string, not escaped because they
        # are not literals
        second_group_items.extend([r"\s", r"\Z"])
        # join them up with a pipe
        second_group = "|".join(second_group_items)
        # build the regular expression
        expr = rf"\A\s*(\S*?)({second_group})"
        self._command_pattern = re.compile(expr)

#### `` terminators `instance-attribute`

    terminators

#### `` multiline_commands `instance-attribute`

    multiline_commands = tuple(multiline_commands) if multiline_commands is not None else ()

#### `` aliases `instance-attribute`

    aliases = dict(aliases) if aliases is not None else {}

#### `` shortcuts `instance-attribute`

    shortcuts = tuple(sorted(items(), key=lambda x: len(x[0]), reverse=True))

#### `` is_valid_command

    is_valid_command(word, *, is_subcommand=False)

Determine whether a word is a valid name for a command.

Commands cannot include redirection characters, whitespace, or termination characters. They also cannot start with a shortcut.

PARAMETER | DESCRIPTION
--- | ---
`word` | the word to check as a command TYPE: `str`
`is_subcommand` | Flag whether this command name is a subcommand name TYPE: `bool` DEFAULT: `False`

RETURNS | DESCRIPTION
--- | ---

`tuple[bool, str]`

a tuple of a boolean and an error string

If word is not a valid command, return `False` and an error string suitable for inclusion in an error message of your choice::

    checkit = ">"
    valid, errmsg = statement_parser.is_valid_command(checkit)
    if not valid:
        errmsg = f"alias: {errmsg}"

Source code in `cmd2/parsing.py`

    345
    346
    347
    348
    349
    350
    351
    352
    353
    354
    355
    356
    357
    358
    359
    360
    361
    362
    363
    364
    365
    366
    367
    368
    369
    370
    371
    372
    373
    374
    375
    376
    377
    378
    379
    380
    381
    382
    383
    384
    385
    386
    387
    388
    389
    390
    391
    392

    def is_valid_command(self, word: str, *, is_subcommand: bool = False) -> tuple[bool, str]:
        """Determine whether a word is a valid name for a command.

        Commands cannot include redirection characters, whitespace,
        or termination characters. They also cannot start with a
        shortcut.

        :param word: the word to check as a command
        :param is_subcommand: Flag whether this command name is a subcommand name
        :return: a tuple of a boolean and an error string

        If word is not a valid command, return ``False`` and an error string
        suitable for inclusion in an error message of your choice::

            checkit = ">"
            valid, errmsg = statement_parser.is_valid_command(checkit)
            if not valid:
                errmsg = f"alias: {errmsg}"
        """
        valid = False

        if not isinstance(word, str):
            return False, f"must be a string. Received {type(word)!s} instead"  # type: ignore[unreachable]

        if not word:
            return False, "cannot be an empty string"

        if word.startswith(constants.COMMENT_CHAR):
            return False, "cannot start with the comment character"

        if not is_subcommand:
            for shortcut, _ in self.shortcuts:
                if word.startswith(shortcut):
                    # Build an error string with all shortcuts listed
                    errmsg = "cannot start with a shortcut: "
                    errmsg += ", ".join(shortcut for (shortcut, _) in self.shortcuts)
                    return False, errmsg

        errmsg = "cannot contain: whitespace, quotes, "

        errchars = (*constants.REDIRECTION_CHARS, *self.terminators)
        errmsg += ", ".join([shlex.quote(x) for x in errchars])

        match = self._command_pattern.search(word)
        if match and word == match.group(1):
            valid = True
            errmsg = ""
        return valid, errmsg

#### `` tokenize

    tokenize(line)

Lex a string into a list of tokens. Shortcuts and aliases are expanded and comments are removed.

PARAMETER | DESCRIPTION
--- | ---
`line` | the command line being lexed TYPE: `str`

RETURNS | DESCRIPTION
--- | ---
`list[str]` | A list of tokens

RAISES | DESCRIPTION
--- | ---
`Cmd2ShlexError` | if a shlex error occurs (e.g. No closing quotation)

Source code in `cmd2/parsing.py`

    394
    395
    396
    397
    398
    399
    400
    401
    402
    403
    404
    405
    406
    407
    408
    409
    410
    411
    412
    413
    414
    415

    def tokenize(self, line: str) -> list[str]:
        """Lex a string into a list of tokens. Shortcuts and aliases are expanded and comments are removed.

        :param line: the command line being lexed
        :return: A list of tokens
        :raises Cmd2ShlexError: if a shlex error occurs (e.g. No closing quotation)
        """
        # expand shortcuts and aliases
        line = self._expand(line)

        # check if this line is a comment
        if line.lstrip().startswith(constants.COMMENT_CHAR):
            return []

        # split on whitespace
        try:
            tokens = shlex_split(line)
        except ValueError as ex:
            raise Cmd2ShlexError(ex) from None

        # custom lexing
        return self.split_on_punctuation(tokens)

#### `` parse

    parse(line)

Tokenize the input and parse it into a cmd2.parsing.Statement object.

Stripping comments, expanding aliases and shortcuts, and extracting output redirection directives.

PARAMETER | DESCRIPTION
--- | ---
`line` | the command line being parsed TYPE: `str`

RETURNS | DESCRIPTION
--- | ---
`Statement` | a new cmd2.parsing.Statement object

RAISES | DESCRIPTION
--- | ---
`Cmd2ShlexError` | if a shlex error occurs (e.g. No closing quotation)

Source code in `cmd2/parsing.py`

    417
    418
    419
    420
    421
    422
    423
    424
    425
    426
    427
    428
    429
    430
    431
    432
    433
    434
    435
    436
    437
    438
    439
    440
    441
    442
    443
    444
    445
    446
    447
    448
    449
    450
    451
    452
    453
    454
    455
    456
    457
    458
    459
    460
    461
    462
    463
    464
    465
    466
    467
    468
    469
    470
    471
    472
    473
    474
    475
    476
    477
    478
    479
    480
    481
    482
    483
    484
    485
    486
    487
    488
    489
    490
    491
    492
    493
    494
    495
    496
    497
    498
    499
    500
    501
    502
    503
    504
    505
    506
    507
    508
    509
    510
    511
    512
    513
    514
    515
    516
    517
    518
    519
    520
    521
    522
    523
    524
    525
    526
    527
    528
    529
    530
    531
    532
    533
    534
    535
    536
    537
    538
    539
    540
    541
    542
    543
    544
    545
    546
    547
    548
    549
    550

    def parse(self, line: str) -> Statement:
        """Tokenize the input and parse it into a [cmd2.parsing.Statement][] object.

        Stripping comments, expanding aliases and shortcuts, and extracting output redirection directives.

        :param line: the command line being parsed
        :return: a new [cmd2.parsing.Statement][] object
        :raises Cmd2ShlexError: if a shlex error occurs (e.g. No closing quotation)
        """
        # handle the special case/hardcoded terminator of a blank line
        # we have to do this before we tokenize because tokenizing
        # destroys all unquoted whitespace in the input
        terminator = ""
        if line[-1:] == constants.LINE_FEED:
            terminator = constants.LINE_FEED

        command = ""
        args = ""

        # lex the input into a list of tokens
        tokens = self.tokenize(line)

        # of the valid terminators, find the first one to occur in the input
        terminator_pos = len(tokens) + 1
        for pos, cur_token in enumerate(tokens):
            for test_terminator in self.terminators:
                if cur_token.startswith(test_terminator):
                    terminator_pos = pos
                    terminator = test_terminator
                    # break the inner loop, and we want to break the
                    # outer loop too
                    break
            else:
                # this else clause is only run if the inner loop
                # didn't execute a break. If it didn't, then
                # continue to the next iteration of the outer loop
                continue
            # inner loop was broken, break the outer
            break

        if terminator:
            if terminator == constants.LINE_FEED:
                terminator_pos = len(tokens) + 1

            # everything before the first terminator is the command and the args
            (command, args) = self._command_and_args(tokens[:terminator_pos])

            # we will set the suffix later
            # remove all the tokens before and including the terminator
            tokens = tokens[terminator_pos + 1 :]
        else:
            (testcommand, testargs) = self._command_and_args(tokens)
            if testcommand in self.multiline_commands:
                # no terminator on this line but we have a multiline command
                # everything else on the line is part of the args
                # because redirectors can only be after a terminator
                command = testcommand
                args = testargs
                tokens = []

        redirector = ""
        redirect_to = ""

        # Find which redirector character appears first in the command
        try:
            pipe_index = tokens.index(constants.REDIRECTION_PIPE)
        except ValueError:
            pipe_index = len(tokens)

        try:
            overwrite_index = tokens.index(constants.REDIRECTION_OVERWRITE)
        except ValueError:
            overwrite_index = len(tokens)

        try:
            append_index = tokens.index(constants.REDIRECTION_APPEND)
        except ValueError:
            append_index = len(tokens)

        # Check if output should be piped to a shell command
        if pipe_index < overwrite_index and pipe_index < append_index:
            redirector = constants.REDIRECTION_PIPE

            # Get the tokens for the pipe command and expand ~ where needed
            pipe_to_tokens = tokens[pipe_index + 1 :]
            utils.expand_user_in_tokens(pipe_to_tokens)

            # Build the pipe command line string
            redirect_to = " ".join(pipe_to_tokens)

            # remove all the tokens after the pipe
            tokens = tokens[:pipe_index]

        # Check for output redirect/append
        elif overwrite_index != append_index:
            if overwrite_index < append_index:
                redirector = constants.REDIRECTION_OVERWRITE
                redirector_index = overwrite_index
            else:
                redirector = constants.REDIRECTION_APPEND
                redirector_index = append_index

            redirect_to_index = redirector_index + 1

            # Check if we are redirecting to a file
            if len(tokens) > redirect_to_index:
                unquoted_path = su.strip_quotes(tokens[redirect_to_index])
                if unquoted_path:
                    redirect_to = utils.expand_user(tokens[redirect_to_index])

            # remove all the tokens after the output redirect
            tokens = tokens[:redirector_index]

        if terminator:
            # whatever is left is the suffix
            suffix = " ".join(tokens)
        else:
            # no terminator, so whatever is left is the command and the args
            suffix = ""
            if not command:
                # command could already have been set, if so, don't set it again
                (command, args) = self._command_and_args(tokens)

        # build the statement
        return Statement(
            args,
            raw=line,
            command=command,
            multiline_command=command in self.multiline_commands,
            terminator=terminator,
            suffix=suffix,
            redirector=redirector,
            redirect_to=redirect_to,
        )

#### `` parse_command_only

    parse_command_only(rawinput)

Identify the command and arguments from raw input.

Partially parse input into a [cmd2.PartialStatement][] object.

The command is identified, and shortcuts and aliases are expanded. Multiline commands are identified, but terminators and output redirection are not parsed.

This method is optimized for completion code and gracefully handles unclosed quotes without raising exceptions.

cmd2.parsing.PartialStatement.args will include all output redirection clauses and command terminators.

Note: Unlike cmd2.parsing.StatementParser.parse, this method preserves internal whitespace within the args. It ensures args has no leading whitespace, and it strips trailing whitespace only if all quotes are closed.

PARAMETER | DESCRIPTION
--- | ---
`rawinput` | the command line as entered by the user TYPE: `str`

RETURNS | DESCRIPTION
--- | ---
`PartialStatement` | a [cmd2.PartialStatement][] object representing the split input

Source code in `cmd2/parsing.py`

    552
    553
    554
    555
    556
    557
    558
    559
    560
    561
    562
    563
    564
    565
    566
    567
    568
    569
    570
    571
    572
    573
    574
    575
    576
    577
    578
    579
    580
    581
    582
    583
    584
    585
    586
    587
    588
    589
    590
    591
    592
    593
    594
    595
    596
    597
    598
    599
    600
    601
    602
    603
    604
    605
    606
    607
    608
    609

    def parse_command_only(self, rawinput: str) -> PartialStatement:
        """Identify the command and arguments from raw input.

        Partially parse input into a [cmd2.PartialStatement][] object.

        The command is identified, and shortcuts and aliases are expanded.
        Multiline commands are identified, but terminators and output
        redirection are not parsed.

        This method is optimized for completion code and gracefully handles
        unclosed quotes without raising exceptions.

        [cmd2.parsing.PartialStatement.args][] will include all output redirection
        clauses and command terminators.

        Note:
            Unlike [cmd2.parsing.StatementParser.parse][], this method
            preserves internal whitespace within the args. It ensures
            args has no leading whitespace, and it strips trailing
            whitespace only if all quotes are closed.

        :param rawinput: the command line as entered by the user
        :return: a [cmd2.PartialStatement][] object representing the split input

        """
        # Expand shortcuts and aliases
        line = self._expand(rawinput)

        command = ""
        args = ""
        match = self._command_pattern.search(line)

        if match:
            # Extract the resolved command
            command = match.group(1)

            # If the command is empty, the input was either empty or started with
            # something like a redirector ('>') or terminator (';').
            if command:
                # args is everything after the command match
                args = line[match.end(1) :].lstrip()

                try:
                    # Check for closed quotes
                    shlex_split(args)
                except ValueError:
                    # Unclosed quote: preserve trailing whitespace for completion context.
                    pass
                else:
                    # Quotes are closed: strip trailing whitespace
                    args = args.rstrip()

        return PartialStatement(
            command=command,
            args=args,
            raw=rawinput,
            multiline_command=command in self.multiline_commands,
        )

#### `` get_command_arg_list

    get_command_arg_list(command_name, to_parse, preserve_quotes)

Retrieve just the arguments being passed to their `do_*` methods as a list.

Convenience method used by the argument parsing decorators.

PARAMETER | DESCRIPTION
--- | ---
`command_name` | name of the command being run TYPE: `str`

`to_parse`

what is being passed to the `do_*` method. It can be one of two types:

1. An already parsed cmd2.Statement
2. An argument string in cases where a `do_*` method is explicitly called. Calling `do_help('alias create')` would cause `to_parse` to be 'alias create'.

In this case, the string will be converted to a cmd2.Statement and returned along with the argument list.

TYPE: `Statement | str`

`preserve_quotes` | if `True`, then quotes will not be stripped from the arguments TYPE: `bool`

RETURNS | DESCRIPTION
--- | ---
`tuple[Statement, list[str]]` | A tuple containing the cmd2.Statement and a list of strings representing the arguments

Source code in `cmd2/parsing.py`

    611
    612
    613
    614
    615
    616
    617
    618
    619
    620
    621
    622
    623
    624
    625
    626
    627
    628
    629
    630
    631
    632
    633
    634
    635
    636
    637
    638
    639
    640
    641

    def get_command_arg_list(
        self, command_name: str, to_parse: Statement | str, preserve_quotes: bool
    ) -> tuple[Statement, list[str]]:
        """Retrieve just the arguments being passed to their ``do_*`` methods as a list.

        Convenience method used by the argument parsing decorators.

        :param command_name: name of the command being run
        :param to_parse: what is being passed to the ``do_*`` method. It can be one of two types:

                             1. An already parsed [cmd2.Statement][]
                             2. An argument string in cases where a ``do_*`` method is
                                explicitly called. Calling ``do_help('alias create')`` would
                                cause ``to_parse`` to be 'alias create'.

                                In this case, the string will be converted to a
                                [cmd2.Statement][] and returned along with
                                the argument list.

        :param preserve_quotes: if ``True``, then quotes will not be stripped from
                                the arguments
        :return: A tuple containing the [cmd2.Statement][] and a list of
                 strings representing the arguments
        """
        # Check if to_parse needs to be converted to a Statement
        if not isinstance(to_parse, Statement):
            to_parse = self.parse(command_name + " " + to_parse)

        if preserve_quotes:
            return to_parse, to_parse.arg_list
        return to_parse, to_parse.argv[1:]

#### `` split_on_punctuation

    split_on_punctuation(tokens)

Further splits tokens from a command line using punctuation characters.

Punctuation characters are treated as word breaks when they are in unquoted strings. Each run of punctuation characters is treated as a single token.

PARAMETER | DESCRIPTION
--- | ---
`tokens` | the tokens as parsed by shlex TYPE: `list[str]`

RETURNS | DESCRIPTION
--- | ---
`list[str]` | a new list of tokens, further split using punctuation

Source code in `cmd2/parsing.py`

    693
    694
    695
    696
    697
    698
    699
    700
    701
    702
    703
    704
    705
    706
    707
    708
    709
    710
    711
    712
    713
    714
    715
    716
    717
    718
    719
    720
    721
    722
    723
    724
    725
    726
    727
    728
    729
    730
    731
    732
    733
    734
    735
    736
    737
    738
    739
    740
    741
    742
    743
    744
    745
    746
    747
    748
    749
    750
    751
    752

    def split_on_punctuation(self, tokens: list[str]) -> list[str]:
        """Further splits tokens from a command line using punctuation characters.

        Punctuation characters are treated as word breaks when they are in
        unquoted strings. Each run of punctuation characters is treated as a
        single token.

        :param tokens: the tokens as parsed by shlex
        :return: a new list of tokens, further split using punctuation
        """
        # Using a set for faster lookups
        punctuation = {*self.terminators, *constants.REDIRECTION_CHARS}

        punctuated_tokens = []

        for cur_initial_token in tokens:
            # Save tokens up to 1 character in length or quoted tokens. No need to parse these.
            if len(cur_initial_token) <= 1 or cur_initial_token[0] in constants.QUOTES:
                punctuated_tokens.append(cur_initial_token)
                continue

            # Iterate over each character in this token
            cur_index = 0
            cur_char = cur_initial_token[cur_index]

            # Keep track of the token we are building
            new_token = ""

            while True:
                if cur_char not in punctuation:
                    # Keep appending to new_token until we hit a punctuation char
                    while cur_char not in punctuation:
                        new_token += cur_char
                        cur_index += 1
                        if cur_index < len(cur_initial_token):
                            cur_char = cur_initial_token[cur_index]
                        else:
                            break

                else:
                    cur_punc = cur_char

                    # Keep appending to new_token until we hit something other than cur_punc
                    while cur_char == cur_punc:
                        new_token += cur_char
                        cur_index += 1
                        if cur_index < len(cur_initial_token):
                            cur_char = cur_initial_token[cur_index]
                        else:
                            break

                # Save the new token
                punctuated_tokens.append(new_token)
                new_token = ""

                # Check if we've viewed all characters
                if cur_index >= len(cur_initial_token):
                    break

        return punctuated_tokens

### `` shlex_split

    shlex_split(str_to_split)

Split the string str_to_split using shell-like syntax.

A wrapper around shlex.split() that uses cmd2's preferred arguments.

This allows other classes to easily call split() the same way StatementParser does.

PARAMETER | DESCRIPTION
--- | ---
`str_to_split` | the string being split TYPE: `str`

RETURNS | DESCRIPTION
--- | ---
`list[str]` | A list of tokens

Source code in `cmd2/parsing.py`

    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39

    def shlex_split(str_to_split: str) -> list[str]:
        """Split the string *str_to_split* using shell-like syntax.

        A wrapper around shlex.split() that uses cmd2's preferred arguments.

        This allows other classes to easily call split() the same way StatementParser does.

        :param str_to_split: the string being split
        :return: A list of tokens
        """
        return shlex.split(str_to_split, comments=False, posix=False)