# Testsuite

`rmlint` has a not yet complete but quite powerful testsuite. It is not complete yet (and probably never will), but it’s already a valuable boost of confidence in `rmlint's` correctness.

The tests are based on `pytest` and are written in `python>=3.0`. Every testcase just runs the (previously built) `rmlint` binary a and parses its json output. So they are technically blackbox-tests.

On every commit, those tests are additionally run on TravisCI.

## Control Variables

The behaviour of the testsuite can be controlled by certain environment variables which are:

- `RM_TS_DIR`: Testdir to create files in. Can be very large with some tests, sometimes `tmpfs` might therefore slow down your computer. By default `/tmp` will be used.

- `RM_TS_USE_VALGRIND`: Run each test inside of valgrind’s memcheck. (slow)

- `RM_TS_CHECK_LEAKS`: Fail test if valgrind indicates (definite) memory leak.

- `RM_TS_USE_GDB`: Run tests inside of `gdb`. Fatal signals will trigger a backtrace.

- `RM_TS_PEDANTIC`: Run each test several times with different optimization options and check for errors between the runs. (slow).

- `RM_TS_SLEEP`: Waits a long time before executing a command. Useful for starting the testcase and manually running `rmlint` on the priorly generated testdir.

- `RM_TS_PRINT_CMD`: Print the command that is currently run.

- `RM_TS_KEEP_TESTDIR`: If a test failed, keep the test files.

Additionally slow tests can be omitted with by appending `-k 'not slow'` to the commandline. More information on this syntax can be found on the pytest documentation.

Before each release we call the testsuite (at least) like this:

    $ sudo RM_TS_USE_VALGRIND=1 RM_TS_PRINT_CMD=1 RM_TS_PEDANTIC=1 pytest -s -a 'not slow and not known_issue'

The `sudo` here is there for executing some tests that need root access (like the creating of bad user and group ids). Most tests will work without.

## Coverage

To see which functions need more testcases we use `gcov` to detect which lines were executed (and how often) by the testsuite. Here’s a short quickstart using `lcov`:

    $ CFLAGS="-fprofile-arcs -ftest-coverage" LDFLAGS="-fprofile-arcs -ftest-coverage" scons -j4 DEBUG=1
    $ sudo RM_TS_USE_VALGRIND=1 RM_TS_PRINT_CMD=1 RM_TS_PEDANTIC=1 pytest -s -a 'slow and not known_issue'
    $ lcov --capture --directory . --output-file coverage.info
    $ genhtml coverage.info --output-directory out

The coverage results are updated from time to time here:

> http://sahib.github.io/rmlint/gcov/index.html

## Structure

    tests
    ├── test_formatters   # Tests for output formatters (like sh or json)
    ├── test_options      # Tests for normal options like --merge-directories etc.
    ├── test_types        # Tests for all lint types rmlint can find
    └── utils.py          # Common utilities shared among tests.

## Templates

A template for a testcase looks like this:

    from tests.utils import *

    def test_basic(usual_setup_usual_teardown):
        create_file('xxx', 'a')
        create_file('xxx', 'b')

        head, *data, footer = run_rmlint('-a city -S a')

        assert footer['duplicate_sets'] == 1
        assert footer['total_lint_size'] == 3
        assert footer['total_files'] == 2
        assert footer['duplicates'] == 1

## Rules

- Test should be able to run as normal user.

- If that’s not possible, check at the beginning of the testcase with this:

      if not runs_as_root():
          return

- Regressions in `rmlint` should get their own testcase so they do not appear again.

- Slow tests can be marked with a slow attribute:

      import pytest

      @pytest.mark.slow
      def test_debian_support(usual_setup_usual_teardown):
          assert random.choice([True, False]):

- Unresolved issues can be marked with the `known_issue` attribute to avoid failing automated travis testing.