Using Python to develop test cases

Author
  Ali Paikan

# Description

Test cases can be written using Python scripting language (e.g., .py) to be loaded and executed as separate plug-ins. This allows to have a simple test runner which loads and run the tests at run time without any needs to recompile the test runner. Using a simple example, we show how this can be done.

Note
  The source code of the tutorial can be also found in the `'examples/python-plugin'` folder within the Robot Testing Framework project directory.

# Requirements

You need the Python development library to use the plug-ins. On most of the Linux distribution it can be easily installed using the package manager (e.g., `$ sudo apt-get install python-dev`). If you use Windows, try Python Releases for Windows. OSX users can also follow the instruction from http://docs.python-guide.org/en/latest/starting/install/osx/.

After installing the Python development package, you need to enable the Python plug-in system in the Robot Testing Framework and recompile it:

       $ cd robot-testing-framework/build
       $ cmake ../ -DENABLE_PYTHON_PLUGIN=ON

# Writing the test case in Python

Writing test cases in Python is simple. All you need is to declare a `TestCase` class and implement your test functional code inside the `run()` method. The `setup()` and `tearDown()` can be also optionally defined to setup or tear down the user defined fixture. By default, the name of the test case is the name of its plug-in file. This cane be changed by using using `setName()`.

    '''
     robottestingframework module is automatically imported by the python plugin loader
     to invoke the corresponding test case methods. To develop a new
     test case simply implement the following class; (setup and tearDown
     methods are optional) :

     class TestCase:
         def setup(self, param):
             return True

         def run(self):

         def tearDown(self):

     The following methods are for reporting, failure or assertions:

     robottestingframework.setName(name)             : sets the test name (defualt is the test filename)
     robottestingframework.testReport(msg)           : reports a informative message
     robottestingframework.testCheck(condition, msg) : reports the test message and marks the test as failed if condition is false
     robottestingframework.assertError(msg)          : throws an error exception with message
     robottestingframework.asserFail(msg)            : throws a failure exception with message
    '''

    class TestCase:
        # setup is called before the test's run to setup
        # the user defined fixture
        # @return Boolean (True/False uppon success or failure)
        def setup(self, param):
            robottestingframework.testReport("Preparing setup...")
            return True

        # The implementation of the test goes here
        def run(self):
            robottestingframework.testCheck(5>3, "5 is bigger than 3.")
            robottestingframework.testCheck(5<3, "5 is smaller than 3.")

        # tearDown is called after the test's run to tear down
        # the user defined fixture
        def tearDown(self):
            robottestingframework.testReport("Tearing down...")

# Writing the test runner

Note
  Basically you do not need to develop a test runner. There is the `robottestingframework-testrunner` utility (see Running test case plug-ins using robottestingframework-testrunner) already implemented in Robot Testing Framework which does the job for you. However, the following example is just for your reference and to understand better how a Python test plug-ing can be loaded and executed from the code.

Your plug-in is ready and can be simply executed using the `robottestingframework-testrunner` utility. However, we show how to use `robottestingframework::PythonPluginLoader` to develop a simple test runner for loading the plug-in and running the test. It is quite simple. First create an instance of `robottestingframework::plugin::PythonPluginLoader` class and call its `open()` method with the plug-in library (i.e., .py) filename as its parameter. The `robottestingframework::plugin::PythonPluginLoader` loads the library and returns a pointer to the `robottestingframework::TestCase` implemented into the plug-in. If the Python plug-in cannot be loaded or it is not an Robot Testing Framework plug-in file, the `robottestingframework::plugin::PythonPluginLoader::open()` returns a null pointer and the error message can be gotten using `robottestingframework::plugin::PythonPluginLoader::getLastError()`. When you have an instance of your test case, it can be used to run the test as usual (see Some examples if you are not familiar with running a test case). In the following example we use a `robottestingframework::TestRunner` to execute our test:

/*
* Robot Testing Framework
*
* Copyright (C) 2015-2019 Istituto Italiano di Tecnologia (IIT)
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
* version 2.1 of the License, or (at your option) any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public
* License along with this library; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
*/
#include <robottestingframework/ConsoleListener.h>
#include <robottestingframework/TestCase.h>
#include <robottestingframework/TestResult.h>
#include <robottestingframework/TestRunner.h>
#include <robottestingframework/python/PythonPluginLoader.h>
#include <stdio.h>
using namespace robottestingframework;
using namespace robottestingframework::plugin;
int main(int argc, char* argv[])
{
if (argc < 2) {
printf("Usage: %s <pyhton plugin file name>\n", argv[0]);
printf("for example: %s mytest.py\n", argv[0]);
return 0;
}
// load the test case plugin
printf("Loading the plugin... \n");
PythonPluginLoader loader;
TestCase* test = loader.open(argv[1]);
if (test == NULL) {
printf("%s\n", loader.getLastError().c_str());
return 0;
}
// create a test listener to collect the result
ConsoleListener listener(false);
// create a test result and add the listeners
TestResult result;
result.addListener(&listener);
// create a test runner and run the test case
TestRunner runner;
runner.addTest(test);
runner.run(result);
return 0;
}
ConsoleListener.h
PythonPluginLoader.h
TestCase.h
TestResult.h
TestRunner.h
robottestingframework::plugin
Definition: PluginLoader.h:29
robottestingframework
Definition: Arguments.h:25

# Building the test runner and run the test

Now you can compile and build the the runner. There is a CMake file in the `examples/python-plugin` folder which helps you to compile and build your simple test runner. Make sure that `RobotTestingFramework_DIR` environment variable is correctly set to point your Robot Testing Framework build or installed directory so that CMake can find the required modules to configure the project.

# Robot Testing Framework
#
# Copyright (C) 2015-2019 Istituto Italiano di Tecnologia (IIT)
#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
# version 2.1 of the License, or (at your option) any later version.
#
# This library is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with this library; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
cmake_minimum_required(VERSION 3.0)
find_package(RobotTestingFramework COMPONENTS DLL)
add_executable(simple_run run.cpp)
target_link_libraries(simple_run RobotTestingFramework::RTF
RobotTestingFramework::RTF_dll)

Now you can build and run the test as follows:

    $ cd examples/python-plugin; mkdir build
    $ cd build; cmake ../; make;
    $ ./simple_run ../mytest.py
    Loading the plugin...
    Starting test runner.
    Test case  started...
    [INFO]  (mytest.lua) reports: Preparing setup...
    [INFO]  (mytest.lua) checks: 5 is bigger than 3
    [FAIL]  (mytest.lua) checks: 5 is less than 3
    [INFO]  (mytest.lua) reports: Tearing down...
    Test case MyTest failed!
    Ending test runner.