# Test Strategy

A key part of the “four opens” is ensuring the OpenStack delivers well-tested and usable software. For more details see: http://docs.openstack.org/project-team-guide/introduction.html#the-four-opens

Experience has shown that untested features are frequently broken, in part due to the velocity of upstream changes. As we aim to ensure we keep all features working across upgrades, we must aim to test all features.

## Reporting Test Coverage

For details on plans to report the current test coverage, refer to Feature Classification.

## Running tests and reporting results

### Running tests locally

To run the unit tests:

    tox -e py3

To run the functional tests:

    tox -e functional

To run a specific test or subset of tests, pass a regex to stestr via tox:

    tox -e py3 -- TestWSGIServer
    tox -e py3 -- nova.tests.unit.compute.test_api

For the full set of options and environment setup details, see https://opendev.org/openstack/nova/src/branch/master/HACKING.rst#running-tests

### Voting in Gerrit

On every review in gerrit, check tests are run on very patch set, and are able to report a +1 or -1 vote. For more details, please see: http://docs.openstack.org/infra/manual/developers.html#automated-testing

Before merging any code, there is an integrate gate test queue, to ensure master is always passing all tests. Nova uses Zuul gating, where each approved change is tested together with other changes ahead of it before merging. For more details, please see: https://zuul-ci.org/docs/zuul/latest/gating.html

Because Nova review series are not squashed before merge, each commit in a series should be independently correct and testable.

### Infra vs Third-Party

Tests that use fully open source components are generally run by the OpenStack Infra teams. Test setups that use non-open technology must be run outside of that infrastructure, but should still report their results upstream.

For more details, please see: http://docs.openstack.org/infra/system-config/third_party.html

### Ad-hoc testing

It is particularly common for people to run ad-hoc tests on each released milestone, such as RC1, to stop regressions. While these efforts can help stabilize the release, as a community we have a much stronger preference for continuous integration testing. Partly this is because we encourage users to deploy master, and we generally have to assume that any upstream commit may already been deployed in production.

## Types of tests

### Unit tests

Unit tests help document and enforce the contract for each component. Without good unit test coverage it is hard to continue to quickly evolve the codebase. The correct level of unit test coverage is very subjective, and as such we are not aiming for a particular percentage of coverage, rather we are aiming for good coverage. Generally, every code change should have a related unit test: https://opendev.org/openstack/nova/src/branch/master/HACKING.rst#creating-unit-tests

### Integration tests

Today, our integration tests involve running the Tempest test suite on a variety of Nova deployment scenarios. The integration job setup is defined in the `.zuul.yaml` file in the root of the nova repository. Jobs are restricted by queue:

- `check`: jobs in this queue automatically run on all proposed changes even with non-voting jobs

- `gate`: jobs in this queue automatically run on all approved changes (voting jobs only)

- `experimental`: jobs in this queue are non-voting and run on-demand by leaving a review comment on the change of “check experimental”

In addition, we have third parties running the tests on their preferred Nova deployment scenario.

### Functional tests

Nova has a set of in-tree functional tests that focus on things that are out of scope for tempest testing and unit testing. Tempest tests run against a full live OpenStack deployment, generally deployed using devstack. At the other extreme, unit tests typically use mock to test a unit of code in isolation. Functional tests don’t run an entire stack, they are isolated to nova code, and have no reliance on external services. They do have a WSGI app, nova services and a database, with minimal stubbing of nova internals.

### Interoperability tests

The Interop Working Group maintains a list that contains a subset of Tempest tests. These are used to verify if a particular Nova deployment’s API responds as expected. For more details, see: https://docs.opendev.org/openinfra/interop/latest/

this page last updated: 2026-08-20 01:33:50

Except where otherwise noted, this document is licensed under Creative Commons Attribution 3.0 License. See all OpenStack Legal Documents.

found an error? report a bug