Go to the latest version of this document.
search in docstrings

Search bar offers the following options:

- Term presence. The below example searches for documents that must contain “foo”, might contain “bar” and must not contain “baz”: `+foo bar -baz`
- Wildcards. The below example searches for documents with words beginning with “foo”: `foo*`
- Search in specific fields. The following search matches all objects in "twisted.mail" that matches “search”: `+qname:twisted.mail.* +search`

  Possible fields: 'name', 'qname' (fully qualified name), 'docstring', and 'kind'. Last two fields are only applicable if "search in docstrings" is enabled.

- Fuzzy matches. The following search matches all documents that have a word within 1 edit distance of “foo”: `foo~1`

Results provided by Lunr.js

# ``twisted`.`persisted`.`_tokenize``

module documentation
(source)

FIXME:https://github.com/twisted/twisted/issues/3843 This can be removed once t.persisted.aot is removed. New code should not make use of this. Tokenization help for Python programs. vendored from https://github.com/python/cpython/blob/6b825c1b8a14460641ca6f1647d83005c68199aa/Lib/tokenize.py Licence: https://docs.python.org/3/license.html tokenize(readline) is a generator that breaks a stream of bytes into Python tokens. It decodes the bytes according to PEP-0263 for determining source file encoding. It accepts a readline-like method which is called repeatedly to get the next line of input (or b"" for EOF). It generates 5-tuples with these members: the token type (see token.py) the token (a string) the starting (row, column) indices of the token (a 2-tuple of ints) the ending (row, column) indices of the token (a 2-tuple of ints) the original line (string) It is designed to match the working of the Python tokenizer exactly, except that it produces COMMENT tokens for comments and gives type OP for all operators. Additionally, all token lists start with an ENCODING token which tells you which encoding was used to decode the bytes stream.

Class | `TokenInfo` | Undocumented
Class | `Untokenizer` | Undocumented
Exception | `StopTokenizing` | Undocumented
Exception | `TokenError` | Undocumented
Function | `any` | Undocumented
Function | `detect_encoding` | The detect_encoding() function is used to detect the encoding that should be used to decode a Python source file. It requires one argument, readline, in the same way as the tokenize() generator.
Function | `generate_tokens` | Tokenize a source reading Python code as unicode strings.
Function | `group` | Undocumented
Function | `main` | Undocumented
Function | `maybe` | Undocumented
Function | `open` | Open a file in read only mode using the encoding detected by detect_encoding().
Function | `tokenize` | The tokenize() generator requires one argument, readline, which must be a callable object which provides the same interface as the readline() method of built-in file objects. Each call to the function should return one line of input as bytes...
Function | `untokenize` | Transform tokens back into Python source code. It returns a bytes object, encoded using the ENCODING token, which is the first token sequence output by tokenize.
Constant | `AMPER` | Undocumented
Constant | `AMPEREQUAL` | Undocumented
Constant | `ASYNC` | Undocumented
Constant | `AT` | Undocumented
Constant | `ATEQUAL` | Undocumented
Constant | `AWAIT` | Undocumented
Constant | `CIRCUMFLEX` | Undocumented
Constant | `CIRCUMFLEXEQUAL` | Undocumented
Constant | `COLON` | Undocumented
Constant | `COLONEQUAL` | Undocumented
Constant | `COMMA` | Undocumented
Constant | `COMMENT` | Undocumented
Constant | `DEDENT` | Undocumented
Constant | `DOT` | Undocumented
Constant | `DOUBLESLASH` | Undocumented
Constant | `DOUBLESLASHEQUAL` | Undocumented
Constant | `DOUBLESTAR` | Undocumented
Constant | `DOUBLESTAREQUAL` | Undocumented
Constant | `ELLIPSIS` | Undocumented
Constant | `ENCODING` | Undocumented
Constant | `ENDMARKER` | Undocumented
Constant | `EQEQUAL` | Undocumented
Constant | `EQUAL` | Undocumented
Constant | `ERRORTOKEN` | Undocumented
Constant | `GREATER` | Undocumented
Constant | `GREATEREQUAL` | Undocumented
Constant | `INDENT` | Undocumented
Constant | `LBRACE` | Undocumented
Constant | `LEFTSHIFT` | Undocumented
Constant | `LEFTSHIFTEQUAL` | Undocumented
Constant | `LESS` | Undocumented
Constant | `LESSEQUAL` | Undocumented
Constant | `LPAR` | Undocumented
Constant | `LSQB` | Undocumented
Constant | `MINEQUAL` | Undocumented
Constant | `MINUS` | Undocumented
Constant | `N_TOKENS` | Undocumented
Constant | `NAME` | Undocumented
Constant | `NEWLINE` | Undocumented
Constant | `NL` | Undocumented
Constant | `NOTEQUAL` | Undocumented
Constant | `NT_OFFSET` | Undocumented
Constant | `NUMBER` | Undocumented
Constant | `OP` | Undocumented
Constant | `PERCENT` | Undocumented
Constant | `PERCENTEQUAL` | Undocumented
Constant | `PLUS` | Undocumented
Constant | `PLUSEQUAL` | Undocumented
Constant | `RARROW` | Undocumented
Constant | `RBRACE` | Undocumented
Constant | `RIGHTSHIFT` | Undocumented
Constant | `RIGHTSHIFTEQUAL` | Undocumented
Constant | `RPAR` | Undocumented
Constant | `RSQB` | Undocumented
Constant | `SEMI` | Undocumented
Constant | `SLASH` | Undocumented
Constant | `SLASHEQUAL` | Undocumented
Constant | `SOFT_KEYWORD` | Undocumented
Constant | `STAR` | Undocumented
Constant | `STAREQUAL` | Undocumented
Constant | `STRING` | Undocumented
Constant | `TILDE` | Undocumented
Constant | `TYPE_COMMENT` | Undocumented
Constant | `TYPE_IGNORE` | Undocumented
Constant | `VBAR` | Undocumented
Constant | `VBAREQUAL` | Undocumented
Variable | `__author__` | Undocumented
Variable | `__credits__` | Undocumented
Variable | `Binnumber` | Undocumented
Variable | `blank_re` | Undocumented
Variable | `Comment` | Undocumented
Variable | `ContStr` | Undocumented
Variable | `cookie_re` | Undocumented
Variable | `Decnumber` | Undocumented
Variable | `Double` | Undocumented
Variable | `Double3` | Undocumented
Variable | `endpats` | Undocumented
Variable | `Expfloat` | Undocumented
Variable | `Exponent` | Undocumented
Variable | `Floatnumber` | Undocumented
Variable | `Funny` | Undocumented
Variable | `Hexnumber` | Undocumented
Variable | `Ignore` | Undocumented
Variable | `Imagnumber` | Undocumented
Variable | `Intnumber` | Undocumented
Variable | `Name` | Undocumented
Variable | `Number` | Undocumented
Variable | `Octnumber` | Undocumented
Variable | `PlainToken` | Undocumented
Variable | `Pointfloat` | Undocumented
Variable | `PseudoExtras` | Undocumented
Variable | `PseudoToken` | Undocumented
Variable | `Single` | Undocumented
Variable | `Single3` | Undocumented
Variable | `single_quoted` | Undocumented
Variable | `Special` | Undocumented
Variable | `String` | Undocumented
Variable | `StringPrefix` | Undocumented
Variable | `tabsize` | Undocumented
Variable | `Token` | Undocumented
Variable | `Triple` | Undocumented
Variable | `triple_quoted` | Undocumented
Variable | `Whitespace` | Undocumented
Function | `_all_string_prefixes` | Undocumented
Function | `_compile` | Undocumented
Function | `_get_normal_name` | Imitates get_normal_name in tokenizer.c.
Function | `_tokenize` | Undocumented

def any(*choices): (source)

Undocumented

def detect_encoding(readline): (source)

The detect_encoding() function is used to detect the encoding that should be used to decode a Python source file. It requires one argument, readline, in the same way as the tokenize() generator.

It will call readline a maximum of twice, and return the encoding used (as a string) and a list of any lines (left as bytes) it has read in.

It detects the encoding from the presence of a utf-8 bom or an encoding cookie as specified in pep-0263. If both a bom and a cookie are present, but disagree, a SyntaxError will be raised. If the encoding cookie is an invalid charset, raise a SyntaxError. Note that if a utf-8 bom is found, 'utf-8-sig' is returned.

If no encoding is specified, then the default of 'utf-8' will be returned.

def generate_tokens(readline): (source)

Tokenize a source reading Python code as unicode strings.

This has the same API as tokenize(), except that it expects the *readline* callable to return str objects instead of bytes.

def group(*choices): (source)

Undocumented

def main(): (source)

Undocumented

def maybe(*choices): (source)

Undocumented

def open(filename): (source)

Open a file in read only mode using the encoding detected by detect_encoding().

def tokenize(readline): (source)

The tokenize() generator requires one argument, readline, which must be a callable object which provides the same interface as the readline() method of built-in file objects. Each call to the function should return one line of input as bytes. Alternatively, readline can be a callable function terminating with StopIteration: readline = open(myfile, 'rb').__next__ # Example of alternate readline The generator produces 5-tuples with these members: the token type; the token string; a 2-tuple (srow, scol) of ints specifying the row and column where the token begins in the source; a 2-tuple (erow, ecol) of ints specifying the row and column where the token ends in the source; and the line on which the token was found. The line passed is the physical line. The first token sequence will always be an ENCODING token which tells you which encoding was used to decode the bytes stream.

def untokenize(iterable): (source)

Transform tokens back into Python source code. It returns a bytes object, encoded using the ENCODING token, which is the first token sequence output by tokenize. Each element returned by the iterable must be a token sequence with at least two elements, a token number and token value. If only two tokens are passed, the resulting output is poor. Round-trip invariant for full input: Untokenized source will match input source exactly Round-trip invariant for limited input: # Output bytes will tokenize back to the input t1 = [tok[:2] for tok in tokenize(f.readline)] newcode = untokenize(t1) readline = BytesIO(newcode).readline t2 = [tok[:2] for tok in tokenize(readline)] assert t1 == t2

AMPER: `int` = (source)

Undocumented

Value

    19

AMPEREQUAL: `int` = (source)

Undocumented

Value

    41

ASYNC: `int` = (source)

Undocumented

Value

    56

AT: `int` = (source)

Undocumented

Value

    49

ATEQUAL: `int` = (source)

Undocumented

Value

    50

AWAIT: `int` = (source)

Undocumented

Value

    55

CIRCUMFLEX: `int` = (source)

Undocumented

Value

    32

CIRCUMFLEXEQUAL: `int` = (source)

Undocumented

Value

    43

COLON: `int` = (source)

Undocumented

Value

    11

COLONEQUAL: `int` = (source)

Undocumented

Value

    53

COMMA: `int` = (source)

Undocumented

Value

    12

COMMENT: `int` = (source)

Undocumented

Value

    61

DEDENT: `int` = (source)

Undocumented

Value

    6

DOT: `int` = (source)

Undocumented

Value

    23

DOUBLESLASH: `int` = (source)

Undocumented

Value

    47

DOUBLESLASHEQUAL: `int` = (source)

Undocumented

Value

    48

DOUBLESTAR: `int` = (source)

Undocumented

Value

    35

DOUBLESTAREQUAL: `int` = (source)

Undocumented

Value

    46

ELLIPSIS: `int` = (source)

Undocumented

Value

    52

ENCODING: `int` = (source)

Undocumented

Value

    63

ENDMARKER: `int` = (source)

Undocumented

Value

    0

EQEQUAL: `int` = (source)

Undocumented

Value

    27

EQUAL: `int` = (source)

Undocumented

Value

    22

ERRORTOKEN: `int` = (source)

Undocumented

Value

    60

GREATER: `int` = (source)

Undocumented

Value

    21

GREATEREQUAL: `int` = (source)

Undocumented

Value

    30

INDENT: `int` = (source)

Undocumented

Value

    5

LBRACE: `int` = (source)

Undocumented

Value

    25

LEFTSHIFT: `int` = (source)

Undocumented

Value

    33

LEFTSHIFTEQUAL: `int` = (source)

Undocumented

Value

    44

LESS: `int` = (source)

Undocumented

Value

    20

LESSEQUAL: `int` = (source)

Undocumented

Value

    29

LPAR: `int` = (source)

Undocumented

Value

    7

LSQB: `int` = (source)

Undocumented

Value

    9

MINEQUAL: `int` = (source)

Undocumented

Value

    37

MINUS: `int` = (source)

Undocumented

Value

    15

N_TOKENS: `int` = (source)

Undocumented

Value

    64

NAME: `int` = (source)

Undocumented

Value

    1

NEWLINE: `int` = (source)

Undocumented

Value

    4

NL: `int` = (source)

Undocumented

Value

    62

NOTEQUAL: `int` = (source)

Undocumented

Value

    28

NT_OFFSET: `int` = (source)

Undocumented

Value

    256

NUMBER: `int` = (source)

Undocumented

Value

    2

OP: `int` = (source)

Undocumented

Value

    54

PERCENT: `int` = (source)

Undocumented

Value

    24

PERCENTEQUAL: `int` = (source)

Undocumented

Value

    40

PLUS: `int` = (source)

Undocumented

Value

    14

PLUSEQUAL: `int` = (source)

Undocumented

Value

    36

RARROW: `int` = (source)

Undocumented

Value

    51

RBRACE: `int` = (source)

Undocumented

Value

    26

RIGHTSHIFT: `int` = (source)

Undocumented

Value

    34

RIGHTSHIFTEQUAL: `int` = (source)

Undocumented

Value

    45

RPAR: `int` = (source)

Undocumented

Value

    8

RSQB: `int` = (source)

Undocumented

Value

    10

SEMI: `int` = (source)

Undocumented

Value

    13

SLASH: `int` = (source)

Undocumented

Value

    17

SLASHEQUAL: `int` = (source)

Undocumented

Value

    39

SOFT_KEYWORD: `int` = (source)

Undocumented

Value

    59

STAR: `int` = (source)

Undocumented

Value

    16

STAREQUAL: `int` = (source)

Undocumented

Value

    38

STRING: `int` = (source)

Undocumented

Value

    3

TILDE: `int` = (source)

Undocumented

Value

    31

TYPE_COMMENT: `int` = (source)

Undocumented

Value

    58

TYPE_IGNORE: `int` = (source)

Undocumented

Value

    57

VBAR: `int` = (source)

Undocumented

Value

    18

VBAREQUAL: `int` = (source)

Undocumented

Value

    42

__author__: `str` = (source)

Undocumented

__credits__: `str` = (source)

Undocumented

Binnumber: `str` = (source)

Undocumented

blank_re = (source)

Undocumented

Comment: `str` = (source)

Undocumented

ContStr = (source)

Undocumented

cookie_re = (source)

Undocumented

Decnumber: `str` = (source)

Undocumented

Double: `str` = (source)

Undocumented

Double3: `str` = (source)

Undocumented

endpats: `dict` = (source)

Undocumented

Expfloat = (source)

Undocumented

Exponent: `str` = (source)

Undocumented

Floatnumber = (source)

Undocumented

Funny = (source)

Undocumented

Hexnumber: `str` = (source)

Undocumented

Ignore = (source)

Undocumented

Imagnumber = (source)

Undocumented

Intnumber = (source)

Undocumented

Name: `str` = (source)

Undocumented

Number = (source)

Undocumented

Octnumber: `str` = (source)

Undocumented

PlainToken = (source)

Undocumented

Pointfloat = (source)

Undocumented

PseudoExtras = (source)

Undocumented

PseudoToken = (source)

Undocumented

Single: `str` = (source)

Undocumented

Single3: `str` = (source)

Undocumented

single_quoted: `set` = (source)

Undocumented

Special = (source)

Undocumented

String = (source)

Undocumented

StringPrefix = (source)

Undocumented

tabsize: `int` = (source)

Undocumented

Token = (source)

Undocumented

Triple = (source)

Undocumented

triple_quoted: `set` = (source)

Undocumented

Whitespace: `str` = (source)

Undocumented

def _all_string_prefixes(): (source)

Undocumented

@functools.lru_cache(None)
def _compile(expr): (source)

Undocumented

def _get_normal_name(orig_enc): (source)

Imitates get_normal_name in tokenizer.c.

def _tokenize(readline, encoding): (source)

Undocumented