Skip to content
Back to top
View this page

# What is this?

Pegen is the parser generator used in CPython to produce the parser used by the interpreter. It allows to produce PEG parsers from a description of a formal Grammar.

# Installing

Install with `pip` or your favorite PyPi package manager.

    pip install pegen

# How to generate a parser

Given a grammar file compatible with `pegen` (you can write your own or start with one in the data directory), you can easily generate a parser by running:

    python -m pegen <path-to-grammar-file> -o parser.py

This will generate a file called `parser.py` in the current directory. This can be used to parse code using the grammar that we just used:

    python parser.py <file-with-code-to-parse>

# Differences with CPython’s Pegen

This repository exists to distribute a version of the Python PEG parser generator used by CPython that can be installed via PyPI, with some improvements. Although the official PEG generator included in CPython can generate both Python and C code, this distribution of the generator only allows to generate Python code. This is due to the fact that the C code generated by the generator included in CPython includes a lot of implementation details and private headers that are not available for general usage.

The official PEG generator for Python 3.9 and later is now included in the CPython repo under Tools/peg_generator/.

See also PEP 617.

# Indices and tables

- About PEG parsers
  - How PEG Parsers Work
    - Key ideas
    - Consequences or the ordered choice operator
    - Grammatical elements and rules
    - Tokenization
    - Memoization
    - Hard and Soft keywords
    - Error handling
  - References
- Syntax
  - Grammar Expressions
    - `# comment`
    - `e1 e2`
    - `e1 | e2`
    - `( e )`
    - `[ e ] or e?`
    - `e*`
    - `e+`
    - `s.e+`
    - `&e`
    - `!e`
    - `~e`
    - `&&e`
  - Keywords
  - Return Value
  - Variables in the Grammar
  - Grammar actions
  - Left recursion
  - Syntax error related rules
  - Customizing the generated parser
    - @class NAME
    - @header
    - @subheader
    - @trailer
  - Style

## Developing

We welcome contributions to `pegen`. Check CONTRIBUTING.md to get an idea of how to contribute to the project.