Everybody knows that tests should run fast because they have to be run often. But even with fast tests, the total run time will increase as the amount of tests increases. A complex system will have a couple of thousand or ten thousand tests, with such amount of tests it is important to be able to identify which tests test which code to be possible to identify the useful tests.
Even then, you will come across a situation where you change or add some code, verify that you didn’t break anything by running the relevant tests only to see that there is a failing test. After you investigate where it went wrong and applied a fix you want to verify the test passes now. This can however be an incremental process that takes multiple iterations.
It becomes important to get instant feedback whether the test still fails or not. It is not sufficient anymore to get a result after a minute or five, which can be the case when running all tests. To speed it up, you want to run only the failing tests before running them all again as a final check. jUnit has a feature that allows you to run the failing tests first, but will still run the tests you are not interested in at the moment.
It becomes a burden when you want to re-run your tests, but the previous run is still busy. jUnit however also has an @Ignore annotation that allows you to prevent jUnit from running a test. Adding such an annotation to each test is however cumbersome, and takes often more time than just running the tests. So we came up with something of our own.
We create a file where we would summarize all the groups and tests and assign them to either run (‘true’) or not (‘false’). Since the same mechanism is used for our integration tests that run on Jenkins, we have a local version that overrides these values.
While this mechanism works great, there were some problems with the existing implementation. First of all did not all testsuite use this mechanism, mostly because it was not that straight forward to implement and often involved just copying a lot of code. Which leads me to the next problem of duplicated code. All tests that did implement it had nearly all the same code. Then there are the differences in which the tests handle the value of the tests, some required the fully qualified name, while others wanted the relative location from the testsuite.
Another limitation of the current mechanism is that it can only filter out testcases and not the tests itself. This means that it is still required to use the @Ignore annotation to filter out specific tests.
I am currently working on making this approach more uniform and easier to user. First of all I have designed an test suite that does all the work for you, meaning all you need to do is extend it and gain the full power of the property filter. This makes it easier to create a new testsuite and enforces uniform behaviour. The testsuite uses a newly made runner that filters out the tests, it does this at the level of the tests instead of the test case.
Another great improvement is the way the properties file is organized. The file itself is no longer required, as the default is to run all tests. The files are no longer selected by summarizing them all but can be identified by a simple glob regex. This enables it easier to select just a couple of test by modifying a few lines:
* = excluded
packageA = included
packageA.testCaseB = excluded
packageA.testCaseB.testC = included
The few lines above mean that only the tests in packageA will run, except those of testCaseB that are not testC. This is much shorter than it used to be where we had to specify each test as false, and those we did want to run as true.
The source code is available at GitHub: https://github.com/ChrisVesters/PropertySuite. I am open for questions and suggestions.