..
      Licensed under the Apache License, Version 2.0 (the "License"); you may
      not use this file except in compliance with the License. You may obtain
      a copy of the License at

          http://www.apache.org/licenses/LICENSE-2.0

      Unless required by applicable law or agreed to in writing, software
      distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
      WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
      License for the specific language governing permissions and limitations
      under the License.

      Convention for heading levels in OVN documentation:

      =======  Heading 0 (reserved for the title in a document)
      -------  Heading 1
      ~~~~~~~  Heading 2
      +++++++  Heading 3
      '''''''  Heading 4

      Avoid deeper levels because they do not render well.

=======
Testing
=======

It is possible to test OVN using both tooling provided with Open
vSwitch and using a variety of third party tooling.

Built-in Tooling
----------------

OVN provides a number of different test suites and other tooling for
validating basic functionality of OVN. Before running any of the tests
described here, you must bootstrap, configure and build OVN as
described in :doc:`/intro/install/general`. You do not need to install
OVN, Open vSwitch or to build or load the kernel module to run these test
suites. You do not need superuser privilege to run these test suites.

Unit Tests
~~~~~~~~~~

OVN includes a suite of self-tests. Before you submit patches
upstream, we advise that you run the tests and ensure that they pass. If you
add new features to OVN, then adding tests for those features will
ensure your features don't break as developers modify other areas of OVN.

To run all the unit tests in OVN, one at a time, run::

    $ make check

This takes under 5 minutes on a modern desktop system.

To run all the unit tests in OVN in parallel, run::

    $ make check TESTSUITEFLAGS=-j8

You can run up to eight threads. This takes under a minute on a modern 4-core
desktop system.

To see a list of all the available tests, run::

    $ make check TESTSUITEFLAGS=--list

To run only a subset of tests, e.g. test 123 and tests 477 through 484, run::

    $ make check TESTSUITEFLAGS='123 477-484'

Tests do not have inter-dependencies, so you may run any subset.

To run tests matching a keyword, e.g. ``ovsdb``, run::

    $ make check TESTSUITEFLAGS='-k ovsdb'

To see a complete list of test options, run::

    $ make check TESTSUITEFLAGS=--help

The results of a testing run are reported in ``tests/testsuite.log``. Report
test failures as bugs and include the ``testsuite.log`` in your report.

.. note::
  Sometimes a few tests may fail on some runs but not others. This is usually a
  bug in the testsuite, not a bug in OVN itself. If you find that a
  test fails intermittently, please report it, since the developers may not
  have noticed. You can make the testsuite automatically rerun tests that fail,
  by adding ``RECHECK=yes`` to the ``make`` command line, e.g.::

      $ make check TESTSUITEFLAGS=-j8 RECHECK=yes

Debugging unit tests
++++++++++++++++++++

To initiate debugging from artifacts generated from `make check` run, set the
``OVS_PAUSE_TEST`` environment variable to 1.  For example, to run test case
139 and pause on error::

  $ OVS_PAUSE_TEST=1 make check TESTSUITEFLAGS='-v 139'

When error occurs, above command would display something like this::

   Set environment variable to use various ovs utilities
   export OVS_RUNDIR=<dir>/ovs/_build-gcc/tests/testsuite.dir/0139
   Press ENTER to continue:

And from another window, one can execute ovs-xxx commands like::

   export OVS_RUNDIR=/opt/vdasari/Developer/ovs/_build-gcc/tests/testsuite.dir/0139
   $ ovs-ofctl dump-ports br0
   .
   .

Once done with investigation, press ENTER to perform cleanup operation.

OVS_PAUSE_TEST=1 only pauses failed tests when run with '-v' option.
Tests run without '-v', or successful tests, are not paused.

.. _testing-coverage:

Coverage
~~~~~~~~

If the build was configured with ``--enable-coverage`` and the ``lcov`` utility
is installed, you can run the testsuite and generate a code coverage report by
using the ``check-lcov`` target::

    $ make check-lcov

All the same options are available via TESTSUITEFLAGS. For example::

    $ make check-lcov TESTSUITEFLAGS='-j8 -k ovn'

.. _testing-valgrind:

Valgrind
~~~~~~~~

If you have ``valgrind`` installed, you can run the testsuite under
valgrind by using the ``check-valgrind`` target::

    $ make check-valgrind

When you do this, the "valgrind" results for test ``<N>`` are reported in files
named ``tests/testsuite.dir/<N>/valgrind.*``.

To test the testsuite of kernel datapath under valgrind, you can use the
``check-kernel-valgrind`` target and find the "valgrind" results under
directory ``tests/system-kmod-testsuite.dir/``.

All the same options are available via TESTSUITEFLAGS.

.. hint::
  You may find that the valgrind results are easier to interpret if you put
  ``-q`` in ``~/.valgrindrc``, since that reduces the amount of output.

Static Code Analysis
~~~~~~~~~~~~~~~~~~~~

Static Analysis is a method of debugging Software by examining code rather than
actually executing it. This can be done through 'scan-build' commandline
utility which internally uses clang (or) gcc to compile the code and also
invokes a static analyzer to do the code analysis. At the end of the build, the
reports are aggregated in to a common folder and can later be analyzed using
'scan-view'.

OVN includes a Makefile target to trigger static code analysis::

    $ ./boot.sh
    $ ./configure CC=clang  # clang
    # or
    $ ./configure CC=gcc CFLAGS="-std=gnu99"  # gcc
    $ make clang-analyze

You should invoke scan-view to view analysis results. The last line of output
from ``clang-analyze`` will list the command (containing results directory)
that you should invoke to view the results on a browser.

Datapath testing
~~~~~~~~~~~~~~~~

OVN includes a suite of tests specifically for datapath functionality.
The datapath tests make some assumptions about the environment.  They
must be run under root privileges on a Linux system with support for
network namespaces.  Make sure no other Open vSwitch instance is
running the test suite.  These tests may take several minutes to
complete, and cannot be run in parallel.

To invoke the datapath testsuite with the OVS userspace datapath,
run::

    $ make check-system-userspace

The results of the userspace testsuite appear in
``tests/system-userspace-testsuite.dir``.

To invoke the datapath testsuite with the OVS kernel datapath, run::

    $ make check-kernel

The results of the kernel testsuite appear in
``tests/system-kmod-testsuite.dir``.

The tests themselves must run as root.  If you do not run ``make`` as
root, then you can specify a program to get superuser privileges as
``SUDO=<program>``, e.g. the following uses ``sudo`` (the ``-E``
option is needed to pass through environment variables)::

    $ make check-system-userspace SUDO='sudo -E'

The testsuite creates and destroys tap devices named ``ovs-netdev``
and ``br0``.  If it is interrupted during a test, then before it can
be restarted, you may need to destroy these devices with commands like
the following::

    $ ip tuntap del dev ovs-netdev mode tap
    $ ip tuntap del dev br0 mode tap

All the features documented under `Unit Tests`_ are available for the
datapath testsuites, except that the datapath testsuites do not
support running tests in parallel.

It is also possible to run `retis`_ capture along with the `check-kernel` tests
by setting `OVS_TEST_WITH_RETIS` environment variable to 'yes'.  This can be
useful for debugging the test cases.  For example, the following command can be
used to run the test 168 under `retis`::

    $ make check-kernel OVS_TEST_WITH_RETIS=yes TESTSUITEFLAGS='168 -d'

After the test is completed, the following data will be available in the test
directory:

* `retis.err` - standard error stream of the `retis collect`.
* `retis.log` - standard output of the `retis collect`, contains all captured
  events in the order they appeared.
* `retis.data` - raw events collected by retis, `retis sort` or other commands
  can be used on this file for further analysis.
* `retis.sorted` - text file containing the output of `retis sort` executed on
  the `retis.data`, for convenience.

Requires retis version 1.5 or newer.  And this also requires OVS submodule to
be built with `--enable-usdt-probes`.

.. _retis: https://github.com/retis-org/retis


Performance testing
~~~~~~~~~~~~~~~~~~~

OVN includes a suite of micro-benchmarks to aid a developer in understanding
the performance impact of any changes that they are making. They can be used to
help to understand the relative performance between two test runs on the same
test machine, but are not intended to give the absolute performance of OVN.

To invoke the performance testsuite, run::

    $ make check-perf

This will run all available performance tests. Some of these tests may be
long-running as they need to build complex logical network topologies. In order
to speed up subsequent test runs, some objects (e.g. the Northbound DB) may be
cached. In order to force the tests to rebuild all these objects, run::

    $ make check-perf TESTSUITEFLAGS="--rebuild"

A typical workflow for a developer trying to improve the performance of OVN
would be the following:

0. Optional: Modify/add a performance test to build the topology that you are
   benchmarking, if required.
1. Run ``make check-perf TESTSUITEFLAGS="--rebuild"`` to generate cached
   databases (and complete a test run). The results of each test run are
   displayed on the screen at the end of the test run but are also saved in the
   file ``tests/perf-testsuite.dir/results``.

.. note::
   This step may take some time depending on the number of tests that are being
   rebuilt, the complexity of the tests and the performance of the test
   machine. If you are only using one test, you can specify the test to run by
   adding the test number to the ``make`` command.
   (e.g. ``make check-perf TESTSUITEFLAGS="--rebuild <test number>"``)

2. Run ``make check-perf`` to measure the performance metric that you are
   benchmarking against. If you are only using one test, you can specify the
   test to run by adding the test number to the ``make`` command.
   (e.g. ``make check-perf TESTSUITEFLAGS="<test number>"``)
3. Modify OVN code to implement the change that you believe will improve the
   performance.
4. Go to Step 2. to continue making improvements.

If, as a developer, you modify a performance test in a way that may change one
of these cached objects, be sure to rebuild the test.

The cached objects are stored under the relevant folder in
``tests/perf-testsuite.dir/cached``.

OVN Upgrade Testing
~~~~~~~~~~~~~~~~~~~

Overview
++++++++

OVN upgrade tests validate that the system continues to function correctly
during rolling upgrades, specifically testing the intermediate state where
ovn-controller is upgraded before ovn-northd and the databases.

The upgrade tests run the system test suite from an older OVN version using
binaries (ovn-controller, ovs-vswitchd, etc.) from the current development
version, ensuring backward compatibility.

Running Upgrade Tests Locally
+++++++++++++++++++++++++++++

Basic usage::

    $ make check-upgrade

This will test upgrades from branch-24.03 (the default base version).

Specify a different base version::

    $ make check-upgrade BASE_VERSION=branch-24.09

Run a specific range of tests::

    $ make check-upgrade BASE_VERSION=branch-25.03 TESTSUITEFLAGS="1-100"

Run only unstable tests::

    $ make check-upgrade UNSTABLE=1 TESTSUITEFLAGS="-k unstable"

Environment Variables
+++++++++++++++++++++

*BASE_VERSION*
  Git branch to use as the base version (default: ``branch-24.03``)

  - branch-24.03: the local repo will be used as the source repo.
  - origin/branch-24.03: the local repo origin is used as the source repo.
  - If branch is not found in local repo, it will be searched in its origin
    (e.g. private github repo or ovn_org repo). If not found in private
    github repo, it will be searched in ovn_org repo.

*TESTSUITEFLAGS*
  Test range to run, using autotest syntax (default: ``1-``, meaning all tests)

  - ``1-100`` - Run tests 1 through 100
  - ``50-`` - Run tests 50 and above
  - ``-k unstable`` - Run tests with 'unstable' keyword

  Additional flags to pass to the testsuite. Use ``-d`` to keep test
  directories on success for debugging.

*UNSTABLE*
  Set to ``1`` to run unstable tests (default: disabled)

How Upgrade Tests Work
++++++++++++++++++++++

The upgrade test workflow:

1. *Save Current Binaries*

   The test framework saves binaries from your current working tree:

   - ``ovn-controller``
   - ``ovs-vswitchd``, ``ovsdb-server``
   - ``ovs-vsctl``, ``ovs-ofctl``, ``ovs-appctl``, ``ovs-dpctl``
   - Flow table definitions from ``controller/lflow.h``

2. *Clone and Checkout Base Version*

   Creates ``upgrade-testsuite.dir/ovn-upgrade-base/`` and checks out the
   specified base version.

3. *Patch Old Tests*

   - Updates hardcoded flow table numbers if tables were renumbered
   - Adds schema compatibility filters to suppress expected warnings
   - Replaces OFTABLE_* m4 macros with current values

4. *Build Base Version*

   Builds the base version twice:

   - With patched ``lflow.h`` to create hybrid ``ovn-debug`` tool
   - With original ``lflow.h`` for proper ``ovn-northd`` and ``ovn-nbctl``

5. *Swap Binaries*

   Replaces the base version's binaries with current versions:

   - Base version: ``ovn-northd``, ``ovn-nbctl`` (test infrastructure)
   - Current version: ``ovn-controller``, ``ovs-vswitchd``, ``ovsdb-server``

6. *Run Tests*

   Executes the system test suite from the base version with the mixed
   binary set.

Interpreting Test Failures
++++++++++++++++++++++++++

Test failures during upgrade testing can indicate:

*Backward Compatibility Issues*
  The new ovn-controller is incompatible with the old northd/databases.
  This is a critical issue that must be fixed before release.

*Flow Generation Changes*
  If flow table contents changed intentionally, the (old) test may need the
  ``TAG_TEST_NOT_UPGRADABLE`` tag.

Debugging Failed Tests
++++++++++++++++++++++

On failure, the test directory is preserved in ``upgrade-testsuite.dir/``.

Check the logs::

    $ upgrade-testsuite.dir/git.log  # Git operations
    $ upgrade-testsuite.dir/build-base.log  # Build output
    $ upgrade-testsuite.dir/ovn-upgrade-base/tests/system-kmod-testsuite.log

Keep test directory for debugging::

    $ make check-upgrade TESTSUITEFLAGS="-d"

Marking Tests as Non-Upgradable
+++++++++++++++++++++++++++++++

Some tests cannot run in upgrade scenarios: tests for features not yet
fully present in the base version.

Mark these tests with the ``TAG_TEST_NOT_UPGRADABLE`` keyword::

    AT_SETUP([test that checks flow details])
    AT_KEYWORDS([TAG_TEST_NOT_UPGRADABLE])
    # ... test code ...
    AT_CLEANUP

These tests will be skipped during upgrade testing but run normally otherwise.

CI Integration
++++++++++++++

Upgrade tests run automatically in GitHub Actions:

*On Schedule (Weekly)*
  - Tests all supported versions.

Implementation Details
++++++++++++++++++++++

Tests are run locally through ``check-upgrade`` Makefile target.
The flow for make check-upgrade is:

- Makefile
- ci/ovn_upgrade_test.py: run_upgrade_workflow, run_tests
- ci/linux-build.sh(TESTSUITE=system-test)
- execute_system_tests "check-kernel" "system-kmod-testsuite.log"
- run_system_tests check-kernel

Through the ci the flow is:

- ci.sh: run_in_container ./.ci/linux-build.sh (TESTSUITE=upgrade-test)
- execute_system_tests "check-upgrade" "system-kmod-testsuite.log"
- run_system_tests check-upgrade
- Back to make check-upgrade-flow.
