In part 1 of my testing practices I already discussed what kind of tests I write, and how they grow naturally. In part 2 I will take a look at the main tool when writing tests: the test framework.
I will discuss the features that a testing framework should have to be able to write good tests, and I will discuss some frameworks that I use and which features they provide. The goal of this post is not to advice a specific test framework, but to identify the general ideas that should be embedded in a test framework such that writing tests becomes much easier.
Over the years I have written plenty of tests, both bad and good ones, it is from this experience that I have come to a set of features that a testing framework needs to aid in writing good tests:
- Failure messages: I have written many tests that would fail with a generic message such as “value is not as expected”. Leaving you completely in the dark when the test would fail. A good failure message should tell you why the test failed.
- Logging: Since not all can be captured by a failure message, tests should also log what they are doing so it is easy to trace back the failure.
- Structure: A test should have a clear structure of the form: setup – test – teardown. Each phase should again be written in a clean, readable way.
- Re-useability: Just as with writing product code, test code should be made re-useable. Either with a library of functions, of with a complete hierarchy.
- Dependencies: In a perfect world you would want your tests to be completely independent, in a real world this is impossible. Simply the fact that your integration tests depends on your unit tests is the proof of this. Therefore we need to be able to define a dependency between tests or test suites.
One of my favorite frameworks for Java, is JUnit. Though the framework is designed for unit tests, I believe that you can use this framework for any type of tests as it is general enough. The only downside of JUnit is that it does not provide all the features I want from a test framework.
- Failure messages: There are build-in messages.
- Logging: JUnit does not provide any automatic logging.
- Structure: A clear structure is provided with the annotations.
- Re-useability: You can write your tests in such a way that you can re-use a lot, but you can also write tests that don’t.
- Dependencies: There is no support for specifying dependencies.
Though JUnit scores pretty good, I often feel the need to build something on top of JUnit to actually create the framework that I want. An example of this is the Gwet framework I discussed before. This is build on top of JUnit simply because it allows us to use the annotations that are so useful.
Gwet did solve some of the missing parts that existed in JUnit, as it did automatic logging, forced the user to write tests in such a way that it could provide better failure messages and re-useability. The only part that is still missing are the relationships between test suites, which would make a nice addition.