Loading...
Searching...
No Matches
Running test case plug-ins using robottestingframework-testrunner

# Contents

- Description
- Running a single test case
- Running all test cases from a given path
- Test suite
- Running a single test suite
- Running multiple test suites
- Monitoring tests from your browser
- Using Robot Testing Framework for stress testing
- The robottestingframework-testrunner options

# Description

The `robottestingframework-testrunner` is a command line tool to load and run multiple tests which has been developed as Robot Testing Framework plug-ins. The test case plug-ins are built as dynamic loadable libraries and depending on the target platform can be found as `*`.so, `*`.dll or *.dylib or using Lua, Python, Ruby and other scripting languages (e.g., .lua, .py, .rb). The `robottestingframework-testrunner` can be used to run a single test case, multiple test cases from a given path, a Test suite or multiple test suites from a given path.

# Running a single test case

To run a single test case plug-in, it is enough to give the name of the corresponding plug-ing file to the `robottestingframework-testrunner` using `--test` option. For example on a Linux machine:

     $ robottestingframework-testrunner --verbose --test libmytest.so

The `robottestingframework-testrunner` follows the platform policy to look for the plug-in file. On a Linux machine, the default folders for the shared libraries and the paths indicated in the ``LD_LIBRARY_PATH`` are searched for the corresponding plug-in. Alternatively, the complete path to the plug-in file can be given to load the corresponding plug-in:

     $ robottestingframework-testrunner --verbose --test ~/my-plugins/libmytest.so

In the similar way you can run a test case which has been developed using Lua, Python, Ruby or other languages:

     $ robottestingframework-testrunner --verbose --test ~/my-plugins/mytest.lua
     $ robottestingframework-testrunner --verbose --test ~/my-plugins/mytest.py
     $ robottestingframework-testrunner --verbose --test ~/my-plugins/mytest.rb

The ``--verbose`` enables the verbose mode which allows that the result of the tests to be also written to the standard out put. By default the result of the test are stored in the ``result`.txt` which can be changed by using ``--output`` option.

# Running all test cases from a given path

Multiple test cases can be easily loaded and run using ``--tests`` option. The expected value for this option is a directory where test case plug-ins are stored.

     $ robottestingframework-testrunner --verbose --tests ~/my-plugins

If the ``--recursive`` switch is given with the ``--tests`` option, the sub-folders are also searched for plug-ins.

# Test suite

By definition, a test suite is a set of test cases which share the same test fixture. In Robot Testing Framework a set of test cases (plug-ins) can be grouped in a test suite using a XML file. Here is an example of test suite:

    <?xml version="1.0" encoding="UTF-8"?>

    <suite name="my suite">
        <description> a simple example </description>
        <environment> --robot simulator </environment>
        <fixture param="--from myfixture.ini"> libfixture.so </fixture>
        <fixture param="--from myfixture2.ini"> libfixture.so </fixture>

        <test param="--from myconf.ini"> libtest1 </test>
        <test type="dll"> libtest1 </test>
        <test param=""> ~/mytest/libtest2.so </test>
        <test repetition="5"> ~/mytest/test3.lua </test>
        <test param=""> ~/mytest/test4.py </test>
        <test param=""> ~/mytest/test5.rb </test>
        ...
    </suite>

The example demonstrates a test suite which includes two test cases (`libmytest1.so` and `libmytest2.so`). An optional string parameter can be also specified for each test using `param` attribute in the `test` tag. ​The parameter string is parsed into (argc/argv) format and will be passed to the test case using robottestingframework::TestCase::setup(int argc, char** argv). Alternatively, the raw value of the parameter can be access in the test plug-in implementation using robottestingframework::TestCase::getParam method.

The environment can be used in the implementation of the test cases to know in which setup (e.g., simulation, real world environment) the test case is employed. The environment value can also be accessed in the test plug-in implementation using robottestingframework::TestCase::getEnvironment method.

If a fixture manager plug-in is specified using `` <fixture> tag, it will be loaded by the `robottestingframework-testrunner` and is set for the current test suite. When suite is executed, the fixture manager will be called automatically to setup and monitor the test fixture which is implemented in the fixture plug-in. (Please see the `examples/fixture-plugin` folder for an example of implementing fixture plug-ins.)

Notice that, for `libtest2.so`, a complete path to the plug-in file is given. Thus, the `robottestingframework-testrunner` does not look into system default paths (e.g., `LD_LIBRARY_PATH`) for the plug-in file and it expects to find the plug-in file using the given path. However, this is not the case for `libtest1.so`.

The `robottestingframework-testrunner` usually checks for the extension of the plug-in file name to recognize the type of the plug-in. There is also possibility to explicitly indicate the type of the plug-in using the `type` attribute. For example if `type="dll"`, then the plug-in test should be given without the extension. Depending on which platform (Windows, Linux or Mac) the test is running, the `robottestingframework-testrunner` looks for the correct (shared library) plug-in name (.dll, .so or .dylib).

# Running a single test suite

A single test suite (XML file) can be run using ``--suite`` option. For example:

     $ robottestingframework-testrunner --verbose --suite ~/my-suites/suite.xml

# Running multiple test suites

Multiple test suites can be loaded and rune using ``--suites`` option. The expected value for this option is a directory where test suites (XML files) are stored.

     $ robottestingframework-testrunner --verbose --suites ~/my-suites

If the ``--recursive`` switch is given with the ``--suites`` option, the sub-folders are also searched for test suites.

# Monitoring tests from your browser

The `robottestingframework-testrunner` uses `robottestingframework::WebProgressListener` to provide a simple web server which allows you to monitor the test's running progress from a browser. To do that, you need to run a test/suite by the `robottestingframework-testrunner` using `--web-reporter` option.

     $ robottestingframework-testrunner --web-reporter --verbose --suites ~/my-suites

Then open a web browser and type `http://127.0.0.1:8080` or the remote address where the `robottestingframework-testrunner` has been launched. You can also change the `robottestingframework-testrunner` default port by using `--web-port <port number>` option.

Notice that, the web reporting functionality may not be available by default and you need to enable it when building the Robot Testing Framework by turning on the `ENABLE_WEB_LISTENER` from the cmake.

# Using Robot Testing Framework for stress testing

For stress testing, it is possible to repeat a test case for desired iterations. This can be easily done by providing the number of the test runs as the parameter (e.g., `--repetition 5`) for the `robottestingframework-testrunner` or setting as the attribute of the test case within a test suite XML file. The `robottestingframework-testrunner` repeats running the test for the desired iterations if the test does not fail. Notice that, only the `robottestingframework::TestCase::run()` will be called repeatedly.

     $ robottestingframework-testrunner --verbose --test ~/my-plugins/mytest.lua --repetition 5

# The robottestingframework-testrunner options

Here is the full list of `robottestingframework-testrunner` options:

    usage: robottestingframework-testrunner [options] ...
    options:
      -t, --test            Runs a single test from the given plugin file. (string [=])
          --tests           Runs multiple tests from the given folder which contains plugins. (string [=])
      -s, --suite           Runs a single test suite from the given XML file. (string [=])
          --suites          Runs multiple test suites from the given folder which contains XML files. (string [=])
      -o, --output          The output file to save the result. Default is result.txt (string [=])
          --output-type     The output file type (text, junit) (string [=text])
      -p, --param           Sets the test case parameters. (Can be used only with --test option.) (string [=])
      -e, --environment     Sets the test case environment. (Can be used only with --test option.) (string [=])
          --repetition      Sets the test run repetition. (Can be used only with --test option.) (j [=0])
      -w, --web-reporter    Enables web reporter
          --web-port        Sets the web reporter server port. (The default port number is 8080.) (i [=8080])
      -r, --recursive       Searches into subfolders for plugins or XML files. (Can be used with --tests or --suites options.)
      -d, --detail          Enables verbose mode of test assertions.
      -v, --verbose         Enables verbose mode.
          --version         Shows version information.
      -?, --help            print this message

Author
  Ali Paikan