# Autopkgtest - Running tests

This document gives an overview how to run tests with autopkgtest. It does not cover each detail, please consult the individual manpages like autopkgtest(1), autopkgtest-schroot(1), etc. for all available options.

Ingredients:

- A source package which defines tests in `debian/tests/`. See README.package-tests for how to define them.

  There are plenty of existing package tests in Debian/Ubuntu which you can use as examples and inspiration, just look for a source package with a `Testsuite: autopkgtest` header, or the automatic test running services in Debian and Ubuntu.

- A location for the source/tests: This can be a local source tree, a local .dsc, or "download with apt-get source".

- Optionally some pre-built binary packages which should be tested.

Finally you need a virtualization server, which creates the environment in which the test runs. Depending on how intrusive the test is this can provide various degrees of isolation, from "run on my local system" (fastest, but unsafe) to "run in a temporary virtual machine" (slowest, but highest possible isolation). These are described in detail below.

# autopkgtest

The `autopkgtest` program is the main program to run tests which gets all these ingredients as arguments, in the following form:

    autopkgtest [options] <source package> [<binary package> ...] -- <virt-server> [<virt-server options>]

# Specifying tests and packages

All possible options are explained in the autopkgtest(1) manpage. This section shows the most common scenarios, with using "mysrc" as source package name. Note that specifying the virtualization server and its options is handled in the following section, and it is independent of specifying tests and packages, so it is merely abbreviated as virt-server here.

- Run tests from the Debian source package in the distribution. This calls `apt-get source mysrc` in the virt-server, thus will use whichever distribution/release that `/etc/apt/sources.list` configures:

  `autopkgtest mysrc --` virt-server

- Run tests from a local source tree, using the binary packages from the distribution. This is useful if you are creating or fixing tests, but the actual binary packages are fine:

  `autopkgtest -B packages/mysrc --` virt-server

  If the source tree is already built, then tests which declare the `build-needed` restriction (see README.package-tests) don't need to re-build the source in the virt-server.

- Build a local source package in the virt-server, then run its tests against the built binaries. This is useful if you need to fix a bug in the actual packages to make the tests succeed:

  `autopkgtest packages/mysrc --` virt-server

- Same as above, but with specifying a built source package instead of a source tree:

  `autopkgtest packages/mysrc_*.dsc --` virt-server

- Test new built binaries with a new source and avoid rebuilding them in virt-server. This is useful if you e. g. update a package to a new version and now want to check whether its tests still succeed:

  `autopkgtest -B packages/*.deb packages/mysrc_*.dsc --` virt-server

- The previous case can be simplified if you have a binary .changes from a previous build:

  `autopkgtest packages/*.changes --` virt-server

- Work on the tests, and run them against already built binary packages. This is useful if you want to incrementally improve your tests, without waiting for package builds. You can do this by referencing your existing binaries (../*.deb), and your local working directory (.) as the source package:

  `autopkgtest -B ../*.deb . --` virt-server

# Output

Unless you specify some options, autopkgtest just writes the logging, test outputs, and test results to stdout/stderr and exits with code 0 on success, or some non-zero code if there were skipped or failed tests or problems with the virt-server. (See autopkgtest(1) for defined codes).

For getting output files you have three choices:

- If you just want the "testname: PASS/FAIL" results, use

  `--summary-file=/path/to/summary.txt`.

- If you want the complete output of autopkgtest in a file, use

  `-l /path/to/test.log` (or the long option `--log-file`)

- If you want the log file, the individual test stdout and stderr output, and built binaries (if any) in a directory, use

  `-o /path/to/test-output/` (or the long option `--output-dir`).

You can also combine these.

# Virtualization server

## schroot

    autopkgtest ... -- schroot schroot-name

Run tests in the specified schroot. You can use mk-sbuild(1) to conveniently create schroots, and run this as normal user if you configured schroot accordingly.

This server is the fastest available that provides "proper" file system isolation and revert, but it does not provide enough isolation for tests that need to start services, reconfigure the network, or open TCP ports which are already open at the host. If your test does not need to do these things this is the recommended server, as schroots are also useful for other tasks like building packages with sbuild.

See autopkgtest-schroot(1) manpage.

## LXC

    autopkgtest ... -- lxc container-name

Run tests in the specified LXC container. Containers provide full service and network isolation, but tests or packages cannot change the kernel or hardware configuration. If your test does not need that, this is the recommended server as it is faster than QEMU and works on all Linux architectures.

`container-name` will be cloned or be called with a temporary overlay file system if you specify the `-e` (`--ephemeral`) option, thus it will never be modified and you can run several tests in parallel safely. Unless your test or architecture or RAM availability doesn't work with overlayfs, using -e is highly recommended for better performance.

If your user can get root privileges with sudo, you can call autopkgtest as your normal user and specify `-s` (`--sudo`) so that the container can be started as root.

See autopkgtest-virt-lxc(1) manpage. This also explains how to build containers.

## QEMU

    autopkgtest ... -- qemu path/to/image

Run tests with QEMU/KVM using the specified image. The image will be run with a temporary overlay file system, thus it will never be modified and you can run several tests in parallel safely.

If your test needs a full machine including kernel/hardware access, this is the recommended runner; it provides complete system isolation, revert and breaks-testbed capabilities. But it is also the one with the biggest overhead and only works well on architectures with KVM acceleration (i. e. mostly x86).

See autopkgtest-virt-qemu(1) manpage. This also explains how to build suitable images, and the requirements of the guest.

## null

    autopkgtest ... -- null

This does not do any virtualization, but runs tests straight on the host. Beware that this will leave some clutter on your system (installed test or build dependency packages, configuration changes that the tests might make, etc.). It is not able to run tests with the "breaks-testbed" restriction. See autopkgtest-virt-null(1) manpage.

## chroot

    autopkgtest ... -- chroot /path/to/chroot

Run tests in the specified chroot. You need to call autopkgtest as root for this. There is no automatic cleanup or revert for the chroot, so unless you can provide this by some other means, don't use this.

## ssh

    autopkgtest ... -- ssh -l joe -H testhost.example.com

This is a generic runner for an externally set up testbed which assumes nothing else than a working ssh connection. This can call a "setup script" to create/configure a testbed (such as spinning up a cloud VM with nova or setting up SSH on a phone through ADB). See the manpage for details. autopkgtest ships setup scripts for an adb host (mostly for Ubuntu Touch), for nova (for cloud instances) and for MaaS currently; see their comment headers in `/usr/share/autopkgtest/ssh-setup/`.

## unshare

    autopkgtest ... -- unshare

Run tests in a chroot tarball. Uses the unshare command to create a chroot and cleans up at the end. This can be used as a normal user, no extra capabilities on the host system are needed. By default it uses tarballs from $XDG_CACHE_HOME/sbuild, like sbuild.

# Using sysvinit

As discussed in Debian bug 844123, it's not entirely trivial to run autopkgtest with sysvinit in a VM because autopkgtest needs a console on ttyS0. The user needs to ensure the VM supports that. When starting from a VM with systemd as init, one can achieve the desired behavior by running autopkgtest with the following command

System Message: ERROR/3 (`doc/README.running-tests.rst`, line 243)

Unexpected indentation.

    --setup-commands 'apt-get install -y --allow-remove-essential sysvinit-core
     systemd-sysv- ; sed -i "/ttyS0/ s/^#//" /etc/inittab'