TUT: C++ Unit Test Framework

Documentation

- TUT How-To
  minimum steps to make TUT work for you

- TUT Design
  what's hidden under the hood of tut.h, and why things are such as they are

- TUT Usage Example
  it's better to see once...

- TUT As Is
  complete source of TUT

Distribution

- The TUT License
  almost no restrictions to use

- TUT Downloads
  latest version of TUT as well as other related stuff

Support

- TUT Frequently Asked Questions
  often asked questions and answers for them

- Links
  related projects and concepts

- TUT Author
  who is the author

TUT Design

In this document I attempt to explain the decisions made while developing TUT.

Requirements

One day I ran into need of unit test framework for C++. So, I've made a small research and discovered C++Unit, boost::test and a bunch of similar libraries.. Though they were usable, I was not satisfied with the approach they offered; so I designed my own variant of unit test framework based on the following restrictions:

- No C-style macros
- No manual registration for test groups and methods
- No libraries of any kind
- Neutrality to user interface
- No Javisms

C-style macros and what's wrong with them

Usually C++ Unit test frameworks define a lot of macroses to achieve the goals other languages have as built-in features: for example, Java is able to show you the whole exception stack; and C++ cannot. So, to achieve the same (or similar) results, C++ frameworks often declare a macro to catch any exception and trace __FILE__ and __LINE__ variables.

The problem is that it turns the C++ code into something that is hard to read, where "hard to read" actually means "hard to maintain".

Macros don't recognize namespace borders, so a simple macro can expand in the user code into something unexpected. To avoid this, we have to give macros unique prefixes, and this, in turn, reduces code readability even more.

From bad to worse, C-style macros can't handle modern C++ templates, so comma separated template arguments will break the macro, since preprocessor will handle the template as two arguments (separated by the comma used in the template) to this macro.

And the final contra for macros is that even if used they cannot achieve the same usability level as the native language tools; for example, macros cannot generate a full stack trace (at least, in a platform-independent manner). So it looks like we loose readability and get almost nothing for this.

See also Bjarne Stroustrup notices about macros harmness: So, what's wrong with using macros?

Manual registration and why it annoys me

In JUnit (Java-based Unit Tests framework) reflection is used to recognize user-written test methods. C++ has no reflection or similar mechanism, so user must somehow tell the framework that "this, this and that" methods should be considered as test methods, and others are just helpers for them.

The same can be said about test groups, which have to be registered in test runner object.

Again, most C++ frameworks introduce macros or at least methods to register a freestanding function or method as a test instance. I find writing redundant code rather annoying: I have to write test itself and then I have to write more code to mark that my test is a test. Even more, if I forget to register the method, nothing will warn me or somehow indicate that I have not done what I should.

Library and integration problems

Most of C++ frameworks require building a library that user must link to test application to be able to run tests. The problem is that various platforms imply different ways for building libraries. One popular C++ framework has more than 60% bugs in their bug database that sound like "cannot build library on platform XXX" or "built library doesn't work on platform YYY".

Besides, some platforms has complexities in library memory management (e.g. Win32).

User interface

Some frameworks provide predefined formatters for output results, for example CSV or XML. This restricts users in test results presentation options. What if a user wants some completely different format? Of course, he can implement his own formatter, but why frameworks provide useless formatters then?

The ideal test framework must do only one thing: run tests. Anything beyond that is the responsibility of the user code. Framework provides the test results, and the user code then represents them in any desired form.

Javisms

Most implementors of C++ test frameworks know about JUnit and inspired by this exciting tool. But, carelessly copying a Java implementation to C++, we can get strange and ugly design.

Rather obvious example: JUnit has methods for setting up a test (setUp) and for cleaning after it (tearDown). I know at least two C++ frameworks that have these methods with the same semantics and names. But in C++ the job these methods do is the responsibility of constructor and destructor! In Java we don't have guaranteed destruction, so JUnit authors had to invent their own replacement for it - tearDown(); and it was natural then to introduce constructing counterpart - setUp(). Doing the same in C++ is absolutely redundant

C++ has its own way of working, and whenever possible, I am going to stay at the C++ roads, and will not repeat Java implementation just because it is really good for Java.

Decisions

No C-style macros

The solution is that simple: just do not use any macros. I personally never needed a macro during development.

No manual registration

Since C++ has no reflection, the only way to mark a method as a test is to give it a kind of predefined name.

There would be a simple solution: create a set of virtual methods in test object base class, and allow user to overwrite them. The code might look like:

     struct a_test_group : public test_group { virtual void test1() { ... } virtual
    			void test2() { ... } };

Unfortunately, this approach has major drawbacks:

- It scales badly. Consider, we have created 100 virtual test methods in a test group, but user needs 200. How can he achieve that? There is no proper way. Frankly speaking, such a demand will arise rarely (mostly in script-generated tests), but even the possibility of it makes this kind of design seemingly poor.
- There is no way to iterate virtual methods automatically. We would end up writing code that calls test1(), then test2(), and so on, each with its own exception handling and reporting.

Another possible solution is to substitute reflection with a dynamic loading. User then would write static functions with predefined names, and TUT would use dlsym()/GetProcAddress() to find out the implemented tests.

But I rejected the solution due to its platform and library operations dependencies. As I described above, the library operations are quite different on various platform.

There was also an idea to have a small parser, that can scan the user code and generate registration procedure. This solution only looks simple; parsing free-standing user code can be a tricky procedure, and might easily overgrow twelve TUTs in complexity.

Fortunately, modern C++ compilers already have a tool that can parse the user code and iterate methods. It is compiler template processing engine. To be more precise, it is template specialization technique.

The following code iterates all methods named test<N> ranging from n to 0, and takes the address of each:

    template <class Test,class Group,int n>
    struct tests_registerer
    {
    	static void reg(Group& group)
    	{
    		group.reg(n,&Test::template	test<n>);
    		tests_registerer<Test,Group,n-1>::reg(group);
    	}
    };

    template<class Test,class Group>
    struct tests_registerer<Test,Group,0>
    {
    	static void reg(Group&){};
    };
    ...
    test_registerer<test,group,100>.reg(grp);

This code generates recursive template instantiations until it reaches tests_registerer<Test,Group,0> which has empty specialization. There the recursion stops.

The code is suitable for our needs because in the specialization preference is given to the user-written code, not to the generic one. Suppose we have a default method test<N> in our test group, and the user-written specialization for test<1>. So while iterating, compiler will get the address of the default method for all N, except 1, since user has supplied a special version of that method.

    template<int N>
    void test() { };
    ...
    template<>
    void test_group::test<1>() { // user code here }

This approach can be regarded as kind of compile-time virtual functions, since the user-written methods replace the default implementation. At the same time, it scales well - one just has to specify another test number upper bound at compile time. The method also allows iteration of methods, keeping code compact.

Library

Since we dig into the template processing, it is natural to not build any libraries, therefor this problem mostly disappeares. Unfortunately, not completely: our code still needs some central point where it could register itself. But that point (singleton) is so small that it would be an overkill to create library just to put there one single object. Instead, we assume that the user code will contain our singleton somewhere in the main module of test application.

User interface. Callbacks.

Our code will perform only minimamum set of tasks: TUT shall run tests. But we still need a way to adapt the end-user presentation requirements. For some of users it would be enough to see only failed tests in listing; others would like to see the complete plain-text report; some would prefer to get XML reports, and some would not want to get any reports at all since they draw their test execution log in GUI plugin for an IDE.

"Many users" means "many demands", and satisfying all of them is quite a hard task. Attempt to use a kind of silver bullet (like XML) is not the right solution, since user would lack XML parser in his environment, or just would not want to put it into his project due to integration complexities.

The decision was made to allow users to form their reports by themselfs. TUT will report an event, and the user code will form some kind of an artefact based on this event.

The implementation of this decision is interface tut::callback. The user code creates a callback object, and passes it to the runner. When an appropriate event occures, the test runner invokes callback methods. User code can do anything, from dumping test results to std::cout to drawing 3D images, if desired.

STL

Initially, there were plans to make TUT traits-based in order not to restrict it with STL only, but have a possibility to use other kinds of strings (TString, CString), containers and intercepted exceptions.

In the current version, these plans are not implemented due to relative complexity of the task. For example, the actual set of operations can be quite different for various map implementations and this makes writing generic code much harder.

Thus so far TUT is completely STL-based, since STL is the only library existing virtually on every platform.