# Loading and Running Tests

To run gabbi tests with a test harness they must be generated in some fashion and then run. This is accomplished by a test loader. Initially gabbi only supported those test harnesses that supported the `load_tests` protocol in UnitTest. It now possible to also build and run tests with pytest with some limitations described below.

Note

It is also possible to run gabbi tests from the command line. See YAML Runner.

Note

By default gabbi will load YAML files using the `safe_load` function. This means only basic YAML types are allowed in the file. For most use cases this is fine. If you need custom types (for example, to match NaN) it is possible to set the `safe_yaml` parameter of `build_tests()` to `False`. If custom types are used, please keep in mind that this can limit the portability of the YAML files to other contexts.

Warning

If test are being run with a runner that supports concurrency (such as `testrepository`) it is critical that the test runner is informed of how to group the tests into their respective suites. The usual way to do this is to use a regular expression that groups based on the name of the yaml files. For example, when using `testrepository` the `.testr.conf` file needs an entry similar to the following:

    group_regex=gabbi\.suitemaker\.(test_[^_]+_[^_]+)

## UnitTest Style Loader

To run the tests with a `load_tests` style loader a test file containing a `load_tests` method is required. That will look a bit like:

    """A sample test module."""

    # For pathname munging
    import os

    # The module that build_tests comes from.
    from gabbi import driver

    # We need access to the WSGI application that hosts our service
    from myapp import wsgiapp

    # We're using fixtures in the YAML files, we need to know where to
    # load them from.
    from myapp.test import fixtures

    # By convention the YAML files are put in a directory named
    # "gabbits" that is in the same directory as the Python test file.
    TESTS_DIR = 'gabbits'

    def load_tests(loader, tests, pattern):
        """Provide a TestSuite to the discovery process."""
        test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
        # Pass "require_ssl=True" as an argument to force all tests
        # to use SSL in requests.
        return driver.build_tests(test_dir, loader,
                                  intercept=wsgiapp.app,
                                  fixture_module=fixtures)

For details on the arguments available when building tests see `build_tests()`.

Once the test loader has been created, it needs to be run. There are many options. Which is appropriate depends very much on your environment. Here are some examples using `unittest` or `testtools` that require minimal knowledge to get started.

By file:

    python -m testtools.run -v test/test_loader.py

By module:

    python -m testttols.run -v test.test_loader

    python -m unittest -v test.test_loader

Using test discovery to locate all tests in a directory tree:

    python -m testtools.run discover

    python -m unittest discover test

See the source distribution and the tutorial repo for more advanced options, including using `testrepository` and `subunit`.

## pytest

Since pytest does not support the `load_tests` system, a different way of generating tests is required. Two techniques are supported.

The original method (described below) used yield statements to generate tests which pytest would collect. This style of tests is deprecated as of `pytest>=3.0` so a new style using pytest fixtures has been developed.

Warning

The pytest loader now requires that `test_loader_name` be set when `gabbi.driver.py_test_generator()` is called.

### pytest >= 3.0

In the newer technique, a test file is created that uses the `pytest_generate_tests` hook. Special care must be taken to always import the `test_pytest` method which is the base test that the pytest hook parametrizes to generate the tests from the YAML files. Without the method, the hook will not be called and no tests generated.

Here is a simple example file:

    """A sample pytest module for pytest >= 3.0."""

    # For pathname munging
    import os

    # The module that py_test_generator comes from.
    from gabbi import driver

    # We need test_pytest so that pytest test collection works properly.
    # Without this, the pytest_generate_tests method below will not be
    # called.
    from gabbi.driver import test_pytest  # noqa

    # We need access to the WSGI application that hosts our service
    from myapp import wsgiapp

    # We're using fixtures in the YAML files, we need to know where to
    # load them from.
    from myapp.test import fixtures

    # By convention the YAML files are put in a directory named
    # "gabbits" that is in the same directory as the Python test file.
    TESTS_DIR = 'gabbits'

    def pytest_generate_tests(metafunc):
        test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
        driver.py_test_generator(
            test_dir, intercept=wsgiapp.app,
            test_loader_name=__name__,
            fixture_module=fixtures, metafunc=metafunc)

This can then be run with the usual pytest commands. For example:

    py.test -svx pytest3.0-example.py

### pytest < 3.0

When using the older technique, test file must be created that calls `py_test_generator()` and yields the generated tests. That will look a bit like this:

    """A sample pytest module."""

    # For pathname munging
    import os

    # The module that build_tests comes from.
    from gabbi import driver

    # We need access to the WSGI application that hosts our service
    from myapp import wsgiapp

    # We're using fixtures in the YAML files, we need to know where to
    # load them from.
    from myapp.test import fixtures

    # By convention the YAML files are put in a directory named
    # "gabbits" that is in the same directory as the Python test file.
    TESTS_DIR = 'gabbits'

    def test_gabbits():
        test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
        # Pass "require_ssl=True" as an argument to force all tests
        # to use SSL in requests.
        test_generator = driver.py_test_generator(
            test_loader_name=__name__,
            test_dir, intercept=wsgiapp.app,
            fixture_module=fixtures)

        for test in test_generator:
            yield test

This can then be run with the usual pytest commands. For example:

    py.test -svx pytest-example.py

The older technique will continue to work with all versions of `pytest<4.0` but `>=3.0` will produce warnings. If you want to use the older technique but not see the warnings add `--disable-pytest-warnings` parameter to the invocation of `py.test`.