# Manuel

Manuel lets you mix and match traditional doctests with custom test syntax.

Several plug-ins are included that provide new test syntax (see Included Functionality). You can also create your own plug-ins.

For example, if you’ve ever wanted to include a large chunk of Python in a doctest but were irritated by all the “>>>” and “…” prompts required, you’d like the `manuel.codeblock` module. It lets you execute code using Sphinx-style “.. code-block:: python” directives. The markup looks like this:

    .. code-block:: python

        import foo

        def my_func(bar):
            return foo.baz(bar)

Incidentally, the implementation of `manuel.codeblock` is only 23 lines of code.

The plug-ins included in Manuel make good examples while being quite useful in their own right. The Manuel documentation makes extensive use of them as well. Follow the “Show Source” link to the left to see the reST source of this document.

For a large example of creating test syntax, take a look at the FIT Table Example or for all the details, Theory of Operation.

To see how to get Manuel wired up see Getting Started.

## Included Functionality

Manuel includes several plug-ins out of the box:

manuel.capture

  stores regions of a document in variables for later processing

manuel.codeblock

  executes code in “.. code-block:: python” blocks

manuel.doctest

  provides traditional doctest processing as a Manuel plug-in

manuel.footnote

  executes code in reST-style footnodes each time they’re referenced (good for getting incidental code out of the main flow of a document)

manuel.ignore

  ignores parts of a document while running tests

manuel.isolation

  makes it easier to have test isolation in doctests

manuel.testcase

  identify parts of tests as individual test cases so they can be run independently

## Getting Started

The plug-ins used for a test are composed together using the “+” operator. Let’s say you wanted a test that used doctest syntax as well as footnotes. You would create a Manuel instance to use like this:

    import manuel.doctest
    import manuel.footnote

    m = manuel.doctest.Manuel()
    m += manuel.footnote.Manuel()

You would then pass the Manuel instance to a `manuel.testing.TestSuite`, including the names of documents you want to process:

    manuel.testing.TestSuite(m, 'test-one.txt', 'test-two.txt')

### Using unittest

The simplest way to get started with Manuel is to use `unittest` to run your tests:

    import manuel.codeblock
    import manuel.doctest
    import manuel.testing
    import unittest

    def test_suite():
        m = manuel.doctest.Manuel()
        m += manuel.codeblock.Manuel()
        return manuel.testing.TestSuite(m, 'test-one.txt', 'test-two.txt')

    if __name__ == '__main__':
        unittest.TextTestRunner().run(test_suite())

### Using zope.testing

If you want to use a more featureful test runner you can use zope.testing’s test runner (usable stand-alone – it isn’t dependent on the Zope application server). Create a file named `tests.py` with a `test_suite()` function that returns a test suite.

The suite can be either a `manuel.testing.TestSuite` object or a `unittest.TestSuite` as demonstrated below.

    import manuel.codeblock
    import manuel.doctest
    import manuel.testing

    def test_suite():
        suite = unittest.TestSuite()

        # here you add your other tests to the suite...

        # now you can add the Manuel tests
        m = manuel.doctest.Manuel()
        m += manuel.codeblock.Manuel()
        suite.addTest(manuel.testing.TestSuite(m,
            'test-one.txt', 'test-two.txt'))

        return suite

### Others

To use another test runner, like nose or pytest:

    import manuel.codeblock
    import manuel.doctest
    import manuel.testing

    m = manuel.doctest.Manuel()
    m += manuel.codeblock.Manuel()
    manueltest = manuel.testing.TestFactory(m)

    class MyTest(unittest.TestCase):

        def setUp(self):
            self.a = 1
            self.globs = dict(c=9)

        test1 = manueltest('doc1.ex')

        @manueltest('doc2.ex')
        def test2(self):
            self.x = 5

        test3 = manueltest('doc3.ex')

Here, we instantiated TestFactory with a Manuel instance to create manueltest, which is a factory for creating Manuel-based tests using on the given Manuel instance. We then used that to create 3 tests.

The first and third tests just execute tests in the named files, doc1.ex and doc3.ex. The class’ setUp method is used to set up the test.

The second test also executes tests in a named file, doc2.ex, but it decorates a function that provides additional setup code that runs after the class setup code.

When tests are run this way:

- The test globals contain the test instance in the test variable.

- If a test case defines a globs attribute, it must be a dictionary and it’s contents are added to the test globals.

### Customizing the TestCase class

Manuel has its own `manuel.testing.TestClass` class that `manuel.testing.TestSuite` uses. If you want to customize it, you can pass in your own class to TestSuite.

     import os.path
     import manuel.testing

     class StripDirsTestCase(manuel.testing.TestCase):
         def shortDescription(self):
             return os.path.basename(str(self))
     suite = manuel.testing.TestSuite(
         m, path_to_test, TestCase=StripDirsTestCase)

    >>> list(suite)[0].shortDescription()
    'bugs.txt'

## Doctests

Manuel is all about making testable documents and well-documented tests. Of course, Python’s doctest module is a long-standing fixture in that space, so it only makes sense for Manuel to support doctest syntax.

Handling doctests is easy:

    import manuel.doctest

    m = manuel.doctest.Manuel()
    suite = manuel.testing.TestSuite(m, 'my-doctest.txt')

Of course you can mix in other Manuel syntax plug-ins as well (including ones you write yourself).

    import manuel.doctest
    import manuel.codeblock

    m = manuel.doctest.Manuel()
    m += manuel.codeblock.Manuel()
    suite = manuel.testing.TestSuite(m, 'my-doctest-with-code-blocks.txt')

The `manuel.doctest.Manuel` constructor also takes `optionflags` and `checker` arguments.

    m = manuel.doctest.Manuel(optionflags=optionflags, checker=checker)

See the doctest documentation for more information about the available options and output checkers

Note

`zope.testing.renormalizing` provides an `OutputChecker` for smoothing out differences between actual and expected output for things that are hard to control (like memory addresses and time). See the module’s documentation for more information on how it works. Here’s a short example that smoothes over the differences between CPython’s and PyPy’s NameError messages:

    import re
    import zope.testing.renormalizing
    checker = zope.testing.renormalizing.RENormalizing([
        (re.compile(r"NameError: global name '([a-zA-Z0-9_]+)' is not defined"),
         r"NameError: name '\1' is not defined"),
    ])

## Capturing Blocks

When writing documentation the need often arises to describe the contents of files or other non-Python information. You may also want to put that information under test. `manuel.capture` helps with that.

For example, if you were writing the problems for a programming contest, you might want to describe the input and output files for each challenge, but you want to be sure that your examples are correct.

To do that you might write your document like this:

    Challenge 1
    ===========

    Write a program that sorts the numbers in a file.

    Example
    -------

    Given this example input file::

        6
        1
        8
        20
        11
        65
        2

    .. -> input

    Your program should generate this output file::

        1
        2
        6
        8
        11
        20
        65

    .. -> output

        >>> input_lines = input.splitlines()
        >>> correct = '\n'.join(map(str, sorted(map(int, input_lines)))) + '\n'
        >>> output == correct
        True

This uses the syntax implemented in `manuel.capture` to capture a block of text into a variable (the one named after “->”).

Whenever a line of the structure “.. -> VAR” is detected, the text of the previous block will be stored in the given variable.

Of course, lines that start with “.. “ are reST comments, so when the document is rendered with docutils or Sphinx, the tests will dissapear and only the intended document contents will remain. Like so:

    Challenge 1
    ===========

    Write a program that sorts the numbers in a file.

    Example
    -------

    Given this example input file::

        6
        1
        8
        20
        11
        65
        2

    Your program should generate this output file::

        1
        2
        6
        8
        11
        20
        65

## Code Blocks

Sphinx and other docutils extensions provide a “code-block” directive, which allows inlined snippets of code in reST documents.

The `manuel.codeblock` module provides the ability to execute the contents of Python code-blocks. For example:

    .. code-block:: python

        print('hello')

If the code-block generates some sort of error…

    .. code-block:: python

        print(does_not_exist)

…that error will be reported:

    >>> document.process_with(m, globs={})
    Traceback (most recent call last):
        ...
    NameError: name 'does_not_exist' is not defined

If you find that you want to include a code-block in a document but don’t want Manuel to execute it, use manuel.ignore to ignore that particular block.

### Docutils Code Blocks

Sphinx and docutils have different ideas of how code blocks should be spelled. Manuel supports the docutils-style code blocks too.

    .. code:: python

        a = 1

Docutils options after the opening of the code block are also allowed:

    .. code:: python
        :class: hidden

        a = 1

### Invisible Code Blocks

At times you’ll want to have a block of code that is executed but not displayed in the rendered document (like some setup for later examples).

When using doctest’s native format (“>>>”) that’s easy to do, you just put the code in a reST comment, like so:

    .. this is some setup, it is hidden in a reST comment

        >>> a = 5
        >>> b = a + 3

However, if you want to include a relatively large chunk of Python, you’d rather use a code-block, but that means that it will be included in the rendered document. Instead, `manuel.codeblock` also understands a variant of the code-block directive that is actually a reST comment: “.. invisible-code-block:: python”:

    .. invisible-code-block:: python

        a = 5
        b = a + 3

Note

The “invisible-code-block” directive will work with either one or two colons. The reason is that reST processers (like docutils and Sphinx) will generate an error for unrecognized directives (like invisible-code-block). Therefore you can use a single colon and the line will be interpreted as a comment instead.

## Footnotes

The `manuel.footnote` module provides an implementation of reST footnote handling, but instead of just plain text, the footnotes can contain any syntax Manuel can interpret including doctests.

    >>> import manuel.footnote
    >>> m = manuel.footnote.Manuel()

Here’s an example of combining footnotes with doctests:

    Here we reference a footnote. [1]_

        >>> x
        42

    Here we reference another. [2]_

        >>> x
        100

    .. [1] This is a test footnote definition.

        >>> x = 42

    .. [2] This is another test footnote definition.

        >>> x = 100

    .. [3] This is a footnote that will never be executed.

        >>> raise RuntimeError('nooooo!')

It is also possible to reference more than one footnote on a single line.

    This line has several footnotes on it. [1]_ [2]_ [3]_

        >>> z
        105

    A little prose to separate the examples.

    .. [1] Do something

        >>> w = 3

    .. [2] Do something

        >>> x = 5

    .. [3] Do something

        >>> y = 7

        >>> z = w * x * y

## Ignoring Blocks

Occasionally the need arises to ignore a block of markup that would otherwise be parsed by a Manuel plug-in.

For example, this document has a code-block that will generate a syntax error:

    The following is invalid Python.

    .. code-block:: python

        def foo:
            pass

We can see that when executed, the SyntaxError escapes.

    >>> import manuel.codeblock
    >>> m = manuel.codeblock.Manuel()
    >>> document.process_with(m, globs={})
    Traceback (most recent call last):
      ...
      File "<memory>:4", line 2
         def foo:
                ^
    SyntaxError: ...

The `manuel.ignore` module provides a way to ignore parts of a document using a directive “.. ignore-next-block”.

Because Manuel plug-ins are executed in the order they are accumulated, we want `manuel.ignore` to be the base Manuel object, with any additional plug-ins added to it.

    import manuel.ignore
    import manuel.doctest
    m = manuel.ignore.Manuel()
    m += manuel.codeblock.Manuel()
    m += manuel.doctest.Manuel()

If we add an ignore marker to the block we don’t want processed…

    The following is invalid Python.

    .. ignore-next-block
    .. code-block:: python

        def foo:
            pass

…the error goes away.

    >>> document.process_with(m, globs={})
    >>> print(document.formatted())

### Ignoring Literal Blocks

Ignoring literal blocks is a little more involved:

    Here is some invalid Python:

    .. ignore-next-block

    ::

       >>> lambda: x=1

## Test Isolation

One of the advantages of unittest over doctest is that the individual tests are isolated from one-another.

In large doctests (like this one) you may want to keep later tests from depending on incidental details of earlier tests, preventing the tests from becoming brittle and harder to change.

Test isolation is one approach to reducing this intra-doctest coupling. The `manuel.isolation` module provides a plug-in to help.

The “.. reset-globs” directive resets the globals in the test:

    We define a variable.

        >>> x = 'hello'

    It is still defined.

        >>> print(x)
        hello

    Now we can reset the globals...

    .. reset-globs

    ...and the name binding will be gone:

        >>> print(x)
        Traceback (most recent call last):
            ...
        NameError: name 'x' is not defined

We can see that after the globals have been reset, the second “print(x)” line raises an error.

Of course, resetting to an empty set of global variables isn’t always what’s wanted. In that case there is a “.. capture-globs” directive that saves a baseline set of globals that will be restored at each reset.

    We define a variable.

        >>> x = 'hello'

    It is still defined.

        >>> print(x)
        hello

    We can capture the currently defined globals:

    .. capture-globs

    Of course capturing the globals doesn't disturb them.

        >>> print(x)
        hello

    Now if we define a new global...

        >>> y = 'goodbye'
        >>> print(y)
        goodbye

    .. reset-globs

    ...it will disappear after a reset.

        >>> print(y)
        Traceback (most recent call last):
            ...
        NameError: name 'y' is not defined

    But the captured globals will still be defined.

        >>> print(x)
        hello

## Identifying Test Cases

If you want parts of a document to be individually accessible as test cases (to be able to run just a particular subset of them, for example), a parser can create a region that marks the beginning of a new test case.

Two ways of identifying test cases are included in `manuel.testcase`:

1. by section headings

2. by explicit “.. test-case: NAME” markers.

### Grouping Tests by Heading

    First Section
    =============

    Some prose.

        >>> print('first test case')

    Some more prose.

        >>> print('still in the first test case')

    Second Section
    ==============

    Even more prose.

        >>> print('second test case')

Given the above document, if you’re using zope.testing’s testrunner (located in bin/test), you could run just the tests in the second section with this command:

    bin/test -t "file-name.txt:Second Section"

Or, exploiting the fact that -t does a regex search (as opposed to a match):

    bin/test -t file-name.txt:Second

### Grouping Tests Explicitly

If you would like to identify test cases separately from sections, you can identify them with a marker:

    First Section
    =============

    The following test will be in a test case that is not individually
    identifiable.

        >>> print('first test case (unidentified)')

    Some more prose.

    .. test-case: first-named-test-case

        >>> print('first identified test case')

    Second Section
    ==============

    The test case markers don't have to immediately proceed a test.

    .. test-case: second-named-test-case

    Even more prose.

        >>> print('second identified test case')

Again, given the above document and zope.testing, you could run just the second set of tests with this command:

    bin/test -t file-name.txt:second-named-test-case

Or, exploiting the fact that -t does a regex search again:

    bin/test -t file-name.txt:second

Even though the tests are individually accessable doesn’t mean that they can’t all be run at the same time:

    bin/test -t file-name.txt

Also, if you create a hierarchy of names, you can run groups of tests at a time. For example, lets say that you append “-important” to all your really important tests, you could then run the important tests for a single document like so:

    bin/test -t 'file-name.txt:.*-important$'

or all the “important” tests no matter what file they are in:

    bin/test -t '-important$'

### Both Methods

You can also combine more than one test case identification method if you want. Here’s an example of building a Manuel stack that has doctests and both flavors of test case identification:

    import manuel.doctest
    import manuel.testcase

    m = manuel.doctest.Manuel()
    m += manuel.testcase.SectionManuel()
    m += manuel.testcase.MarkerManuel()

## Further Reading

- Theory of Operation
- FIT Table Example
- Fixed Bugs
- MyST markdown