Loading...
Searching...
No Matches
Using plug-in to develop test cases

Author
  Ali Paikan

# Description

Test cases can be written as dynamic loadable libraries (e.g., .so, .dll) to be loaded and executed as separate plug-ins. This allows to have a simple test runner which loads and run the tests at run time without any needs to recompile the test runner. Using a simple example, we show how this can be done.

Note
  The source code of the tutorial can be also found in the `'examples/plugin'` folder within the Robot Testing Framework project directory. To compile it, you just need to enable the `BUILD_EXAMPLES=ON` flag of the CMake (if it is not already enabled) and build the Robot Testing Framework.

# Requirements

You do not need any external library to use the plug-ins. All you need is to simply enable the plug-in system in the Robot Testing Framework and recompile it if it is not already enabled (it should be enabled by default).

       $ cd robot-testing-framework/build
       $ cmake ../ -DENABLE_PLUGIN=ON

# Writing the plug-in

Create a class which inherited from the TestCase class. Override the `run()` method and/or `setup(int argc, char** argv)` , `tearDown()` if needed. The parameters set for the test case using `robottestingframework::TestCase::setParam` are parsed into standard ```(int argc, char**argv)` format and are passed to the `setup` method. The original string parameter is also accessible using `robottestingframework::TestCase::getParam`. Here is an example:

/*
* Robot Testing Framework
*
* Copyright (C) 2015-2019 Istituto Italiano di Tecnologia (IIT)
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
* version 2.1 of the License, or (at your option) any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public
* License along with this library; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
*/
#ifndef ROBOTTESTINGFRAMEWORK_MYTEST_H
#define ROBOTTESTINGFRAMEWORK_MYTEST_H
#include <robottestingframework/TestCase.h>
class MyTest : public robottestingframework::TestCase
{
public:
MyTest();
bool setup(int argc, char** argv) override;
void tearDown() override;
void run() override;
};
#endif // ROBOTTESTINGFRAMEWORK_MYTEST_H
TestCase.h
robottestingframework::TestCase
The base class to implememnt a test case.
Definition TestCase.h:42
robottestingframework::TestCase::tearDown
virtual void tearDown()
tearDown is called after the test run
robottestingframework::TestCase::run
virtual void run()=0
run is called by the TestCase class if setup is successfull;
robottestingframework::TestCase::setup
virtual bool setup(int argc, char **argv)
setup is called before the test run.

Implement the test body in the `run()` method and add use the `'PREPARE_PLUGIN(...)'` macro to prepare the plug-in. The macro implements the necessary functionalities to make your test class be accessible from a shared library.

/*
* Robot Testing Framework
*
* Copyright (C) 2015-2019 Istituto Italiano di Tecnologia (IIT)
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
* version 2.1 of the License, or (at your option) any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public
* License along with this library; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
*/
#include "MyTest.h"
#include <robottestingframework/TestAssert.h>
#include <robottestingframework/dll/Plugin.h>
using namespace robottestingframework;
ROBOTTESTINGFRAMEWORK_PREPARE_PLUGIN(MyTest)
MyTest::MyTest() :
TestCase("MyTest")
{
}
bool MyTest::setup(int argc, char** argv)
{
ROBOTTESTINGFRAMEWORK_TEST_REPORT("running MyTest::setup...");
return true;
}
void MyTest::tearDown()
{
ROBOTTESTINGFRAMEWORK_TEST_REPORT("running MyTest::teardown...");
// assert an arbitray error for example.
ROBOTTESTINGFRAMEWORK_ASSERT_ERROR("this is just for example!");
}
void MyTest::run()
{
ROBOTTESTINGFRAMEWORK_TEST_REPORT("testing integers");
ROBOTTESTINGFRAMEWORK_TEST_FAIL_IF_FALSE(2 < 3, "is not smaller");
int a = 5;
int b = 3;
ROBOTTESTINGFRAMEWORK_TEST_FAIL_IF_FALSE(a < b, Asserter::format("%d is not smaller than %d.", a, b));
}
Plugin.h
ROBOTTESTINGFRAMEWORK_PREPARE_PLUGIN
#define ROBOTTESTINGFRAMEWORK_PREPARE_PLUGIN(classname)
Definition Plugin.h:30
TestAssert.h
ROBOTTESTINGFRAMEWORK_ASSERT_ERROR
#define ROBOTTESTINGFRAMEWORK_ASSERT_ERROR(message)
Assertion with throwing error exception.
Definition TestAssert.h:90
ROBOTTESTINGFRAMEWORK_TEST_FAIL_IF_FALSE
#define ROBOTTESTINGFRAMEWORK_TEST_FAIL_IF_FALSE(condition, message)
Conditional failure report.
Definition TestAssert.h:149
ROBOTTESTINGFRAMEWORK_TEST_REPORT
#define ROBOTTESTINGFRAMEWORK_TEST_REPORT(message)
Reporting a message to the TestResult.
Definition TestAssert.h:131
robottestingframework::Asserter::format
static std::string format(const char *msg,...)
format can be used to format data to in the same way that sprintf formats.
robottestingframework
Definition Arguments.h:25

# Writing the test runner

Note
  Basically you do not need to develop a test runner. There is the `robottestingframework-testrunner` utility (see Running test case plug-ins using robottestingframework-testrunner) already implemented in Robot Testing Framework which does the job for you. However, the following example is just for your reference and to understand better how a test plug-ing can be loaded and executed from the code.

Your plug-in is ready and can be compiled and built. Before doing, that we show how to use `robottestingframework::DllPluginLoader` to develop a simple test runner for loading the plug-in and running the test. It is quite simple. First create an instance of `robottestingframework::plugin::DllPluginLoader` class and call its `open()` method with the plug-in library (.dll, .so, ...) filename as its paramter. The `robottestingframework::plugin::DllPluginLoader` loads the library and return a pointer to the `robottestingframework::TestCase` implemented into the plug-in. If the plug-in cannot be loaded or it is not an Robot Testing Framework plug-in file, the `robottestingframework::plugin::DllPluginLoader::open()` returns a null pointer and the error message can be gotten using `robottestingframework::plugin::DllPluginLoader::getLastError()`. When you have an instance of your test case, it can be used to run the test as usual (see Some examples if you are not familiar with running a test case). In the following example we use a `robottestingframework::TestRunner` to execute our test:

/*
* Robot Testing Framework
*
* Copyright (C) 2015-2019 Istituto Italiano di Tecnologia (IIT)
*
* This library is free software; you can redistribute it and/or
* modify it under the terms of the GNU Lesser General Public
* License as published by the Free Software Foundation; either
* version 2.1 of the License, or (at your option) any later version.
*
* This library is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* Lesser General Public License for more details.
*
* You should have received a copy of the GNU Lesser General Public
* License along with this library; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
*/
#include <robottestingframework/ConsoleListener.h>
#include <robottestingframework/TestCase.h>
#include <robottestingframework/TestResult.h>
#include <robottestingframework/TestRunner.h>
#include <robottestingframework/dll/DllPluginLoader.h>
#include <stdio.h>
using namespace robottestingframework;
using namespace robottestingframework::plugin;
int main(int argc, char* argv[])
{
if (argc < 2) {
printf("Usage: %s <plugin file name>\n", argv[0]);
printf("for example: %s libmytest.so\n", argv[0]);
return 0;
}
// load the test case plugin
printf("Loading the plugin... \n");
DllPluginLoader loader;
TestCase* test = loader.open(argv[1]);
if (test == NULL) {
printf("%s\n", loader.getLastError().c_str());
return 0;
}
// create a test listener to collect the result
ConsoleListener listener(false);
// create a test result and add the listeners
TestResult result;
result.addListener(&listener);
// create a test runner and run the test case
TestRunner runner;
runner.addTest(test);
runner.run(result);
return 0;
}
ConsoleListener.h
DllPluginLoader.h
TestResult.h
TestRunner.h
robottestingframework::ConsoleListener
class ConsoleListener listens to any messages reported by the tests during the test run,...
Definition ConsoleListener.h:39
robottestingframework::TestResult
The TestResult class is used to deliver the test results including any error and failures produced by...
Definition TestResult.h:44
robottestingframework::TestResult::addListener
void addListener(TestListener *listener)
Adding a new listener.
robottestingframework::TestRunner
The TestRunner class runs the tests added as TestCase or TestSuite.
Definition TestRunner.h:42
robottestingframework::TestRunner::run
void run(TestResult &result)
Run all the tests in the list.
robottestingframework::TestRunner::addTest
void addTest(Test *test)
Adding a new test.
robottestingframework::plugin::DllPluginLoader
The DllPluginLoader loads an test case plug-in and gives the direct access to the TestCase.
Definition DllPluginLoader.h:38
robottestingframework::plugin::DllPluginLoader::open
TestCase * open(const std::string filename) override
open Loads a test case plugin
robottestingframework::plugin::DllPluginLoader::getLastError
std::string getLastError() override
getLastError gets the last error if any.
robottestingframework::plugin
Definition PluginLoader.h:29

# Build and run the test

Now you can compile and build the plug-in and the runner. There is a CMake file in the `examples/plugin` folder which helps you to compile and build your test plugin and the simple test runner. Make sure that `RobotTestingFramework_DIR` environment variable is correctly set to point your Robot Testing Framework build or installed directory so that CMake can find the required modules to configure the project.

# Robot Testing Framework
#
# Copyright (C) 2015-2019 Istituto Italiano di Tecnologia (IIT)
#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
# License as published by the Free Software Foundation; either
# version 2.1 of the License, or (at your option) any later version.
#
# This library is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
# Lesser General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public
# License along with this library; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
cmake_minimum_required(VERSION 3.5)
find_package(RobotTestingFramework COMPONENTS DLL)
add_library(mytest MODULE MyTest.cpp MyTest.h)
target_link_libraries(mytest RobotTestingFramework::RTF
RobotTestingFramework::RTF_dll)
add_executable(simple_run run.cpp)
target_link_libraries(simple_run RobotTestingFramework::RTF
RobotTestingFramework::RTF_dll)

Now you can build and run the test as follows:

    $ cd examples/plugin; mkdir build
    $ cd build; cmake ../; make;
    $ ./simple_run libmytest.so
     Loading the plugin...
     Starting test runner.
     Test case MyTest started...
     [INFO]  (MyTest) reports: running MyTest::setup...
     [INFO]  (MyTest) reports: testing integers
     [FAIL]  (MyTest) checking (a<b): 5 is not smaller than 3.
     [INFO]  (MyTest) reports: running MyTest::teardown...
     [ERROR] (MyTest) asserts error with exception: this is just for example!
     Test case MyTest failed!
     Ending test runner.