Sphinx

# Adding a reference domain

The objective of this tutorial is to illustrate roles, directives and domains. Once complete, we will be able to use this extension to describe a recipe and reference that recipe from elsewhere in our documentation.

Note

This tutorial is based on a guide first published on opensource.com and is provided here with the original author’s permission.

## Overview

We want the extension to add the following to Sphinx:

- A `recipe` directive, containing some content describing the recipe steps, along with a `:contains:` option highlighting the main ingredients of the recipe.

- A `ref` role, which provides a cross-reference to the recipe itself.

- A `recipe` domain, which allows us to tie together the above role and domain, along with things like indices.

For that, we will need to add the following elements to Sphinx:

- A new directive called `recipe`

- New indexes to allow us to reference ingredient and recipes

- A new domain called `recipe`, which will contain the `recipe` directive and `ref` role

## Prerequisites

We need the same setup as in the previous extensions. This time, we will be putting out extension in a file called `recipe.py`.

Here is an example of the folder structure you might obtain:

    └── source
        ├── _ext
        │   └── recipe.py
        ├── conf.py
        └── index.rst

## Writing the extension

Open `recipe.py` and paste the following code in it, all of which we will explain in detail shortly:

      1from collections import defaultdict
      2
      3from docutils.parsers.rst import directives
      4
      5from sphinx import addnodes
      6from sphinx.application import Sphinx
      7from sphinx.directives import ObjectDescription
      8from sphinx.domains import Domain, Index
      9from sphinx.roles import XRefRole
     10from sphinx.util.nodes import make_refnode
     11from sphinx.util.typing import ExtensionMetadata
     12
     13
     14class RecipeDirective(ObjectDescription):
     15    """A custom directive that describes a recipe."""
     16
     17    has_content = True
     18    required_arguments = 1
     19    option_spec = {
     20        'contains': directives.unchanged_required,
     21    }
     22
     23    def handle_signature(self, sig, signode):
     24        signode += addnodes.desc_name(text=sig)
     25        return sig
     26
     27    def add_target_and_index(self, name_cls, sig, signode):
     28        signode['ids'].append('recipe' + '-' + sig)
     29        if 'contains' in self.options:
     30            ingredients = [x.strip() for x in self.options.get('contains').split(',')]
     31
     32            recipes = self.env.get_domain('recipe')
     33            recipes.add_recipe(sig, ingredients)
     34
     35
     36class IngredientIndex(Index):
     37    """A custom index that creates an ingredient matrix."""
     38
     39    name = 'ingredient'
     40    localname = 'Ingredient Index'
     41    shortname = 'Ingredient'
     42
     43    def generate(self, docnames=None):
     44        content = defaultdict(list)
     45
     46        recipes = {
     47            name: (dispname, typ, docname, anchor)
     48            for name, dispname, typ, docname, anchor, _ in self.domain.get_objects()
     49        }
     50        recipe_ingredients = self.domain.data['recipe_ingredients']
     51        ingredient_recipes = defaultdict(list)
     52
     53        # flip from recipe_ingredients to ingredient_recipes
     54        for recipe_name, ingredients in recipe_ingredients.items():
     55            for ingredient in ingredients:
     56                ingredient_recipes[ingredient].append(recipe_name)
     57
     58        # convert the mapping of ingredient to recipes to produce the expected
     59        # output, shown below, using the ingredient name as a key to group
     60        #
     61        # name, subtype, docname, anchor, extra, qualifier, description
     62        for ingredient, recipe_names in ingredient_recipes.items():
     63            for recipe_name in recipe_names:
     64                dispname, typ, docname, anchor = recipes[recipe_name]
     65                content[ingredient].append((
     66                    dispname,
     67                    0,
     68                    docname,
     69                    anchor,
     70                    docname,
     71                    '',
     72                    typ,
     73                ))
     74
     75        # convert the dict to the sorted list of tuples expected
     76        content = sorted(content.items())
     77
     78        return content, True
     79
     80
     81class RecipeIndex(Index):
     82    """A custom index that creates an recipe matrix."""
     83
     84    name = 'recipe'
     85    localname = 'Recipe Index'
     86    shortname = 'Recipe'
     87
     88    def generate(self, docnames=None):
     89        content = defaultdict(list)
     90
     91        # sort the list of recipes in alphabetical order
     92        recipes = self.domain.get_objects()
     93        recipes = sorted(recipes, key=lambda recipe: recipe[0])
     94
     95        # generate the expected output, shown below, from the above using the
     96        # first letter of the recipe as a key to group thing
     97        #
     98        # name, subtype, docname, anchor, extra, qualifier, description
     99        for _name, dispname, typ, docname, anchor, _priority in recipes:
    100            content[dispname[0].lower()].append((
    101                dispname,
    102                0,
    103                docname,
    104                anchor,
    105                docname,
    106                '',
    107                typ,
    108            ))
    109
    110        # convert the dict to the sorted list of tuples expected
    111        content = sorted(content.items())
    112
    113        return content, True
    114
    115
    116class RecipeDomain(Domain):
    117    name = 'recipe'
    118    label = 'Recipe Sample'
    119    roles = {
    120        'ref': XRefRole(),
    121    }
    122    directives = {
    123        'recipe': RecipeDirective,
    124    }
    125    indices = {
    126        RecipeIndex,
    127        IngredientIndex,
    128    }
    129    initial_data = {
    130        'recipes': [],  # object list
    131        'recipe_ingredients': {},  # name -> object
    132    }
    133    data_version = 0
    134
    135    def get_full_qualified_name(self, node):
    136        return f'recipe.{node.arguments[0]}'
    137
    138    def get_objects(self):
    139        yield from self.data['recipes']
    140
    141    def resolve_xref(self, env, fromdocname, builder, typ, target, node, contnode):
    142        match = [
    143            (docname, anchor)
    144            for name, sig, typ, docname, anchor, prio in self.get_objects()
    145            if sig == target
    146        ]
    147
    148        if len(match) > 0:
    149            todocname = match[0][0]
    150            targ = match[0][1]
    151
    152            return make_refnode(builder, fromdocname, todocname, targ, contnode, targ)
    153        else:
    154            print('Awww, found nothing')
    155            return None
    156
    157    def add_recipe(self, signature, ingredients):
    158        """Add a new recipe to the domain."""
    159        name = f'recipe.{signature}'
    160        anchor = f'recipe-{signature}'
    161
    162        self.data['recipe_ingredients'][name] = ingredients
    163        # name, dispname, type, docname, anchor, priority
    164        self.data['recipes'].append((
    165            name,
    166            signature,
    167            'Recipe',
    168            self.env.docname,
    169            anchor,
    170            0,
    171        ))
    172
    173
    174def setup(app: Sphinx) -> ExtensionMetadata:
    175    app.add_domain(RecipeDomain)
    176
    177    return {
    178        'version': '0.1',
    179        'parallel_read_safe': True,
    180        'parallel_write_safe': True,
    181    }

Let’s look at each piece of this extension step-by-step to explain what’s going on.

The directive class

The first thing to examine is the `RecipeDirective` directive:

     1class RecipeDirective(ObjectDescription):
     2    """A custom directive that describes a recipe."""
     3
     4    has_content = True
     5    required_arguments = 1
     6    option_spec = {
     7        'contains': directives.unchanged_required,
     8    }
     9
    10    def handle_signature(self, sig, signode):
    11        signode += addnodes.desc_name(text=sig)
    12        return sig
    13
    14    def add_target_and_index(self, name_cls, sig, signode):
    15        signode['ids'].append('recipe' + '-' + sig)
    16        if 'contains' in self.options:
    17            ingredients = [x.strip() for x in self.options.get('contains').split(',')]
    18
    19            recipes = self.env.get_domain('recipe')
    20            recipes.add_recipe(sig, ingredients)

Unlike Extending syntax with roles and directives and Extending the build process, this directive doesn’t derive from `docutils.parsers.rst.Directive` and doesn’t define a `run` method. Instead, it derives from `sphinx.directives.ObjectDescription` and defines `handle_signature` and `add_target_and_index` methods. This is because `ObjectDescription` is a special-purpose directive that’s intended for describing things like classes, functions, or, in our case, recipes. More specifically, `handle_signature` implements parsing the signature of the directive and passes on the object’s name and type to its superclass, while `add_target_and_index` adds a target (to link to) and an entry to the index for this node.

We also see that this directive defines `has_content`, `required_arguments` and `option_spec`. Unlike the `TodoDirective` directive added in the previous tutorial, this directive takes a single argument, the recipe name, and an option, `contains`, in addition to the nested reStructuredText in the body.

The index classes

Todo

Add brief overview of indices

     1class IngredientIndex(Index):
     2    """A custom index that creates an ingredient matrix."""
     3
     4    name = 'ingredient'
     5    localname = 'Ingredient Index'
     6    shortname = 'Ingredient'
     7
     8    def generate(self, docnames=None):
     9        content = defaultdict(list)
    10
    11        recipes = {
    12            name: (dispname, typ, docname, anchor)
    13            for name, dispname, typ, docname, anchor, _ in self.domain.get_objects()
    14        }
    15        recipe_ingredients = self.domain.data['recipe_ingredients']
    16        ingredient_recipes = defaultdict(list)
    17
    18        # flip from recipe_ingredients to ingredient_recipes
    19        for recipe_name, ingredients in recipe_ingredients.items():
    20            for ingredient in ingredients:
    21                ingredient_recipes[ingredient].append(recipe_name)
    22
    23        # convert the mapping of ingredient to recipes to produce the expected
    24        # output, shown below, using the ingredient name as a key to group
    25        #
    26        # name, subtype, docname, anchor, extra, qualifier, description
    27        for ingredient, recipe_names in ingredient_recipes.items():
    28            for recipe_name in recipe_names:
    29                dispname, typ, docname, anchor = recipes[recipe_name]
    30                content[ingredient].append((
    31                    dispname,
    32                    0,
    33                    docname,
    34                    anchor,
    35                    docname,
    36                    '',
    37                    typ,
    38                ))
    39
    40        # convert the dict to the sorted list of tuples expected
    41        content = sorted(content.items())
    42
    43        return content, True

     1class RecipeIndex(Index):
     2    """A custom index that creates an recipe matrix."""
     3
     4    name = 'recipe'
     5    localname = 'Recipe Index'
     6    shortname = 'Recipe'
     7
     8    def generate(self, docnames=None):
     9        content = defaultdict(list)
    10
    11        # sort the list of recipes in alphabetical order
    12        recipes = self.domain.get_objects()
    13        recipes = sorted(recipes, key=lambda recipe: recipe[0])
    14
    15        # generate the expected output, shown below, from the above using the
    16        # first letter of the recipe as a key to group thing
    17        #
    18        # name, subtype, docname, anchor, extra, qualifier, description
    19        for _name, dispname, typ, docname, anchor, _priority in recipes:
    20            content[dispname[0].lower()].append((
    21                dispname,
    22                0,
    23                docname,
    24                anchor,
    25                docname,
    26                '',
    27                typ,
    28            ))
    29
    30        # convert the dict to the sorted list of tuples expected
    31        content = sorted(content.items())
    32
    33        return content, True

Both `IngredientIndex` and `RecipeIndex` are derived from `Index`. They implement custom logic to generate a tuple of values that define the index. Note that `RecipeIndex` is a simple index that has only one entry. Extending it to cover more object types is not yet part of the code.

Both indices use the method `Index.generate()` to do their work. This method combines the information from our domain, sorts it, and returns it in a list structure that will be accepted by Sphinx. This might look complicated but all it really is is a list of tuples like `('tomato', 'TomatoSoup', 'test', 'rec-TomatoSoup',...)`. Refer to the domain API guide for more information on this API.

These index pages can be referenced with the `ref` role by combining the domain name and the index `name` value. For example, `RecipeIndex` can be referenced with `:ref:`recipe-recipe`` and `IngredientIndex` can be referenced with `:ref:`recipe-ingredient``.

The domain

A Sphinx domain is a specialized container that ties together roles, directives, and indices, among other things. Let’s look at the domain we’re creating here.

     1class RecipeDomain(Domain):
     2    name = 'recipe'
     3    label = 'Recipe Sample'
     4    roles = {
     5        'ref': XRefRole(),
     6    }
     7    directives = {
     8        'recipe': RecipeDirective,
     9    }
    10    indices = {
    11        RecipeIndex,
    12        IngredientIndex,
    13    }
    14    initial_data = {
    15        'recipes': [],  # object list
    16        'recipe_ingredients': {},  # name -> object
    17    }
    18    data_version = 0
    19
    20    def get_full_qualified_name(self, node):
    21        return f'recipe.{node.arguments[0]}'
    22
    23    def get_objects(self):
    24        yield from self.data['recipes']
    25
    26    def resolve_xref(self, env, fromdocname, builder, typ, target, node, contnode):
    27        match = [
    28            (docname, anchor)
    29            for name, sig, typ, docname, anchor, prio in self.get_objects()
    30            if sig == target
    31        ]
    32
    33        if len(match) > 0:
    34            todocname = match[0][0]
    35            targ = match[0][1]
    36
    37            return make_refnode(builder, fromdocname, todocname, targ, contnode, targ)
    38        else:
    39            print('Awww, found nothing')
    40            return None
    41
    42    def add_recipe(self, signature, ingredients):
    43        """Add a new recipe to the domain."""
    44        name = f'recipe.{signature}'
    45        anchor = f'recipe-{signature}'
    46
    47        self.data['recipe_ingredients'][name] = ingredients
    48        # name, dispname, type, docname, anchor, priority
    49        self.data['recipes'].append((
    50            name,
    51            signature,
    52            'Recipe',
    53            self.env.docname,
    54            anchor,
    55            0,
    56        ))

There are some interesting things to note about this `recipe` domain and domains in general. Firstly, we actually register our directives, roles and indices here, via the `directives`, `roles` and `indices` attributes, rather than via calls later on in `setup`. We can also note that we aren’t actually defining a custom role and are instead reusing the `sphinx.roles.XRefRole` role and defining the `sphinx.domains.Domain.resolve_xref` method. This method takes two arguments, `typ` and `target`, which refer to the cross-reference type and its target name. We’ll use `target` to resolve our destination from our domain’s `recipes` because we currently have only one type of node.

Moving on, we can see that we’ve defined `initial_data`. The values defined in `initial_data` will be copied to `env.domaindata[domain_name]` as the initial data of the domain, and domain instances can access it via `self.data`. We see that we have defined two items in `initial_data`: `recipes` and `recipe_ingredients`. Each contains a list of all objects defined (i.e. all recipes) and a hash that maps a canonical ingredient name to the list of objects. The way we name objects is common across our extension and is defined in the `get_full_qualified_name` method. For each object created, the canonical name is `recipe.<recipename>`, where `<recipename>` is the name the documentation writer gives the object (a recipe). This enables the extension to use different object types that share the same name. Having a canonical name and central place for our objects is a huge advantage. Both our indices and our cross-referencing code use this feature.

The `setup` function

As always, the `setup` function is a requirement and is used to hook the various parts of our extension into Sphinx. Let’s look at the `setup` function for this extension.

    1def setup(app: Sphinx) -> ExtensionMetadata:
    2    app.add_domain(RecipeDomain)
    3
    4    return {
    5        'version': '0.1',
    6        'parallel_read_safe': True,
    7        'parallel_write_safe': True,
    8    }

This looks a little different to what we’re used to seeing. There are no calls to `add_directive()` or even `add_role()`. Instead, we have a single call to `add_domain()` followed by some initialization of the standard domain. This is because we had already registered our directives, roles and indexes as part of the directive itself.

## Using the extension

You can now use the extension throughout your project. For example:

index.rst

    Joe's Recipes
    =============

    Below are a collection of my favourite recipes. I highly recommend the
    :recipe:ref:`TomatoSoup` recipe in particular!

    .. toctree::

       tomato-soup

tomato-soup.rst

    The recipe contains `tomato` and `cilantro`.

    .. recipe:recipe:: TomatoSoup
       :contains: tomato, cilantro, salt, pepper

       This recipe is a tasty tomato soup, combine all ingredients
       and cook.

The important things to note are the use of the `:recipe:ref:` role to cross-reference the recipe actually defined elsewhere (using the `:recipe:recipe:` directive).

## Further reading

For more information, refer to the docutils documentation and Sphinx API.

If you wish to share your extension across multiple projects or with others, check out the Third-party extensions section.