# Tests

This tutorial shows how the unit test library can be used to write and run tests.

The code presented in this tutorial is available in the directory `root/tutorials/unit` in the Storm release. You can run it by passing the `-t` flag to Storm: `storm -t tutorials.unit`

## Setup

First, we need to create somewhere to work. For this tutorial, it is enough to create a file `unit.bs` somewhere on your system. The file can be empty for the time being.

After creating the file, open a terminal and change to the directory where you created the file. Then run it by typing:

    storm unit.bs -t unit

If done correctly, Storm should print the message `Passed all 0 tests` and exit. Note that based on how you have installed Storm, you might need to modify the command-line slightly.

The first part of the command line (`unit.bs`) tells Storm to load the file `unit.bs` into the package `unit`, as we have done in previous tutorials. By default, this will cause Storm to look for a function named `main` in the loaded code. However, the `-t` flag overrides this behavior. The flag `-t unit` asks Storm to run all unit tests in the `unit` package that was just loaded, instead of attempting to run the function `main`.

## Writing Tests

To make things more interesting, let's start by writing a small function to test. For simplicity, we will use the iterative implementation of the fibonacci function below:

    Int fibonacci(Int n) {
        Int current = 1;
        Int previous = 1;

        for (Int i = 1; i < n; i++) {
            Int next = current + previous;
            previous = current;
            current = next;
        }

        return current;
    }

Some readers may already have noticed that the implementation above does indeed not work as we would expect. Let's create some tests to illustrate this!

The unit test library is built around creating tests. Each test is essentially a function that may contain `check` statements. A test is defined using the `test` keyword, followed by a name for the test. To be able to use this syntax in Basic Storm, the `test` library must first be included. As such, we add the use statement to the top of our file:

    use test;

And then we can define a suite towards the bottom of the file:

    test Fibonacci {
        check fibonacci(1) == 1;
        check fibonacci(2) == 1;
        check fibonacci(3) == 2;
    }

Now, we can run the test using `storm unit.bs -t unit` to see if our implementation works as we expect. Sadly, this is not the case as Storm produces the following output:

    Running Fibonacci...
    Failed: fibonacci(2) == 1 ==> 2 == 1
    Failed: fibonacci(3) == 2 ==> 3 == 2
    Passed 1 of 3 tests
    Failed 2 tests!

The output ends with a summary that states that only one of our three check statements succeeded, and that 2 failed. Above, we see the lines `Failed:` that indicate which tests failed. These lines both start by reproducing the expression that was tested. Then it outputs a long arrow (`==>`), followed by a representation of what the left- and right-hand side of the operator evaluated to. In the case of `fibonacci(2) == 1`, the left-hand side was `2` and the right-hand side was `1`. This expansion is to make it easier to see why the test failed.

In our case, the reason is that the `fibonacci` implementation is "shifted" by one position. That is, `fibonacci(0)` evaluates `fibonacci(1)` and so on. We can fix this by modifying the initialization of `previous` as follows:

    Int fibonacci(Int n) {
        Int current = 1;
        Int previous = 0;

        for (Int i = 1; i < n; i++) {
            Int next = current + previous;
            previous = current;
            current = next;
        }

        return current;
    }

With this modification, Storm will now print:

    Passed all 3 tests

## Testing for Exceptions

Let's assume that we wish to extend our implementation of `fibonacci` to also support the case where `n` is zero, and to properly reject negative inputs.

Note: A better way to clearly signal that negative inputs are invalid would be to change the function to accept `Nat` instead of `Int`. There are, however, cases where this is not as easy as for the case of `fibonacci`, so we accept `Int` as input to illustrate how to use exceptions in the test library.

We can easy add a check statement for the case where `n` is zero as follows:

    test Fibonacci {
        check fibonacci(0) == 0;
        check fibonacci(1) == 1;
        check fibonacci(2) == 1;
        check fibonacci(3) == 2;
    }

The test currently fails, but we can easily fix it by adding a special case for zero as follows:

    Int fibonacci(Int n) {
        if (n == 0) {
            return 0;
        }

        Int current = 1;
        Int previous = 0;

        for (Int i = 1; i < n; i++) {
            Int next = current + previous;
            previous = current;
            current = next;
        }

        return current;
    }

In the case where `n` is below zero, we expect `fibonacci` to throw the exception `NotSupported`. We can test for this case using the keyword `throws` in a test statement as follows:

    test Fibonacci {
        check fibonacci(0) == 0;
        check fibonacci(1) == 1;
        check fibonacci(2) == 1;
        check fibonacci(3) == 2;

        check fibonacci(-1) throws NotSupported;
    }

If we run the test currently, we will see that the new test fails with the following output:

    Running Fibonacci...
    Failed: fibonacci(-1) ==> did not throw NotSupported as expected.
    Passed 4 of 5 tests
    Failed 1 test!

As we can see above, the message follows a similar structure to before, but here it prints that `NotSupported` was not thrown rather than expanding the expression. We can fix the issue by adding an appropriate check to the start of our `fibonacci` function:

    Int fibonacci(Int n) {
        if (n == 0) {
            return 0;
        } else if (n < 0) {
            throw NotSupported("n must not be negative");
        }

        Int current = 1;
        Int previous = 0;

        for (Int i = 1; i < n; i++) {
            Int next = current + previous;
            previous = current;
            current = next;
        }

        return current;
    }

And with this change, all tests pass again.

## Running Tests Programmatically

So far we have used the `-t` flag on the command line to run the tests. It is of course possible to interact with the tests programmatically as well.

Each test is represented as a regular function in the name tree. As such, we can simply run the function to run the tests inside of it:

    void main() {
        TestResult result = Fibonacci();
        print("Failed: ${result.failed}");
        print(result.toS);
    }

As can be seen above, test suites return an instance of the `TestResult` class that contains information about the tests that were executed. It contains member variables that count the number of tests executed, the number of failures, and so on. The code above prints the number of failed tests, and then prints the string representation of the entire `result` object. If we run the program using `storm unit.bs` (i.e. without `-t unit`), the `main` function will be executed, and we will see the following output:

    Failed: 0
    Passed all 5 tests

As we see from the output, the string representation of the `TestResult` object is what was printed by the test library automatically. As such, it is quite easy to implement custom logic for running individual test suites.

It is also possible to run sets of test suites automatically by using the `runTests` function provided by the library. The function accepts the name of a package that contains the tests that we wish to execute, and optionally also parameters specifying the verbosity of the run. Since the function accepts a `Package` instance, it is convenient to use the `named{}` construct from `lang:bs:macro` to get the current package:

    void main() {
        TestResult = runTests(named{});
        print("Failed: ${result.failed}");
    }

If we run this code, we can see that the `runTests` function produces some additional status messages and prints the result automatically by default. This can of course be disabled by passing `false` as the second parameter to the function:

    Running Fibonacci...
    Passed all 5 tests
    Failed: 0

## Final Notes

It is worth noting that `check` statements can appear anywhere inside a test function. This means that it is possible to use loops and if-statements to programmatically determine which checks should be executed. We could, for example, do the following to verify that `fibonacci` throws an exception for many negative numbers if we wished to (the benefit is, however, debatable):

    test Negative {
        for (Int i = -1; i > -10; i--) {
            check fibonacci(i) throws NotSupported;
        }
    }

In other cases, the code in a test might realize that the implementation is too broken to even execute tests. In situations like this, it is possible to use the `abort;` keyword to abort the test altogether.

Finally, it is worth noting that it is not necessary to place the tests in the same file as the code that is being tested. As with most things in Storm, the test suite can be placed anywhere that is convenient. As such, it is often beneficial to place tests in a sub-package named `tests`. This makes it so that Storm does not have to compile the tests whenever the tested code is used. Test cases can even be structured into a hierarcy of packages, where each package represents a suite of tests. In such cases, the `-t` flag has to be replaced by `-T` in order to instruct the system to traverse packages recursively.