In the previous post, I have been examining the C++ build tools that are available. As a good Software Engineer, the next step of course is to choose a testing framework. From my experience with Java, I have a couple of requirements on what the testing framework should allow. Most of these requirements come from using jUnit and Mockito, which I think are both outstanding frameworks that allow you to easily test your code.
For my tests I examine the following libraries: gTest, CppUTest, Boost, CxxTest and FakeIt. Some important features I am looking for is easy testing/mocking, readable tests, and the limitations of the framework.
gTest, and gMock are very easy to build and use. When creating tests with gTest the test result of each tests is shown, which for big real world applications may become too cluttered. The good thing is that all failed tests are repeated at the bottom, such that you can easily find the failed tests, and at the same time verify whether a specific test has run and passed by doing a quick text search. While you may think there is no reason to show tests that passed, on multiple occasions I did verify whether a test was running or not. There can be different reasons why a test would not be executed, and it is always nice to have an easy way to check this.
gMock is the mocking part of the library, but it has quite a lot of limitations. It requires quite a few changes on your production code to be able to mock it. First of all, you need to mark each method as virtual (or use templated code). Moreover, you have to create every mock yourself. This will become a very cumbersome task and will require a lot of maintenance. Also the style on how to create the mocks is very artificial.
CppUTest was already much harder to get the lib for. For some reason my build did only create the libraries when I installed it to my locale user lib folder. But after this running it went smoothly. A difference with gTest is that passed tests don’t get logged which means there is no clutter. You can however run in verbose mode, to have similar output as gTest, with the only difference that failed tests don’t get repeated at the end. This does make finding failed tests more difficult, but searching for “Failure in TEST” should do the job for you.
CppUMock has the same limitations as gMock. Methods need to be virtual, and you need to override each method. Just like gMock this requires a lot of manual work and will require a lot of maintenance to keep it working.
Boost Test was a big failure for me. Perhaps because of my limited experience with Boost, and C++ in general, because I wasn’t able to link my application. It always complained about undefined symbols.
CxxTest uses an external application to generate cpp files from header files. While this may simplify the way to write tests, having an external application requires a special setup and prepossessing of that file. To me this feels like a pretty artificial way of working and may complicate your build process. But none of that matters, because my tests didn’t work, all of them passed even though I explicitly made one such that it should fail.
FakeIt only offers mocking capabilities and relies on a different library for the other testing features, I have tried FakeIt together with gTest. It is immediately clear that FakeIt is different from the other frameworks in that sense that is gives more a Mockito feeling as it is the only framework that creates the mock for you. There is however still the limitation that the method needs to be marked as virtual.
To conclude, we first have to acknowledge that C++ is not the same as Java, so it makes sense we have different limitations. Specifying every method as virtual, is actually the default behaviour in Java, where you have to explicitly mark a method as final. If you do this however, you can no longer mock calls to this object with Mockito (PowerMock can be a solution here).
For some of my dream features you would require more integration with the compiler to handle the calls. This is probably not what we want either, because that would bind your framework with a specific compiler. For now FakeIt together with gTest would be my preferred choice of testing setup.
If you use some other framework, or have more experience with the frameworks I mentioned here, please let me know. I am eager to learn more about these frameworks and why/how you are using them.