Factory Acceptance Tests

The product we create is a combination of hardware and software, and recently we have introduced the policy of doing a factory acceptance test (FAT) for every vehicle (AGV) that leaves our company. The FAT is meant to make sure that every vehicle that leaves the company works correctly, at least that is the theory because in practice we still have a lot of problems with getting everything working on the customer’s site. The goal of this blog post is to go into depth about the reasons for doing a FAT, the pros and cons and why it is failing for us.

Continue reading

Advertisements

Pre/Post Conditions

Crucial to good Object Orientation, is the separation of concerns and capsulation. This means your objects need a clear API that limits the way the internal state can be changed. The object itself must be responsible for the state and must guard its integrity. While the API defines what operations are allowed, these methods have restrictions of their own. It is essential that the parameters provided to these methods are within the limitations. The only solid way of checking these values is by using preconditions.

Continue reading

Quality Over Quantity

When writing software we often are only focused on just that: getting the feature done. With the rise of Test Driven Development (TDD) some of us will first write tests before the actual code and this is great. Personally I like the concept of TDD, but I can’t seem to make myself adhere to it as I am always to eager to write the code instead of first write the test.

Another reason I don’t do TDD is that I find it hard to write a test for something that doesn’t even exist yet. At least the public interface should exist, but this interface can change often during my implementation because I don’t come up with a full design in the beginning.

But even though I don’t do TDD, I spend a lot of time writing unit tests, making sure all of my code is being covered and that I didn’t forget anything. I do this because this is the only way to avoid bugs in the future. Others however will spend either much less time on writing tests or none at all but will write a lot more code. To an outsider it may seem like these developers are more productive as they output more code. There are however a couple of problems with this idea.

Continue reading

Care About Your Environment

For those who are confused about the title, I am not going to talk about our planet (although it is essential we take care of it). The environment I am referring to is of course the development, testing, build and production environment. All these environments can be different and in most cases they are. There are a couple of reasons for this:

  • You have different customers all having different hardware and software in production you have to integrate with.
  • There is a difference between your company and the customer in setup, which would be very expensive to change.
  • Having a full production environment for development is almost always too expensive.

So unless you are willing to change all of your environments and pay a serious price for it, your environments will differ. And while these differences can be cumbersome and make software development harder, a lot can be done to overcome most of these problems.

Continue reading

Testing Edge Compatibility

Windows 10 is gaining popularity and so does Edge, the new browser that comes with it. With this increasing popularity it becomes important to verify the compatibility of your software with the new platform. Recently I was asked to verify that our Software behaved correctly when using it with Microsoft Edge. While it would have been possible and easy to just do some manual checks, this would not suffice and will not guarantee future changes to be compatible with Edge.

But since we already have automated tests for other browsers, it makes sense to just add Edge as a browser on which these tests have to run. Trying some new technology can be challenging, and often problems arise. Trying to get our Selenium tests running with Edge was no exception.

Continue reading

Testing Your Tests With Mutation Testing

As mentioned in the previous post, I have examined using mutation testing as a way to verify that everything is being tested. In this blog post I will share my experience and the conclusion of this experiment. Mutation testing is a way to test your tests and be confident that bugs will be caught. This is achieved by performing mutations on your code (to change the behaviour) and running all (relevant) tests again to check that at least one has failed. If all tests passed then the mutation was not caught and a test is missing. For my experiment I have used PIT, the eclipse plugin to be more precise.

Continue reading

Code Coverage: Not A Goal, But A Means

The main purpose of testing is to verify code works as expected. Not just now, but in the future as well. It gives us confidence we didn’t break anything and no bugs are introduced. Writing tests is therefore a crucial part of software development that prevents us from having to do a lot of bug fixing after each change.

It is important that nearly all of the code is being tested, in a complex project with thousands or millions of lines of code. How do we know we have enough tests? Just counting the amount of tests and setting a required ration on the lines of codes will not work as some lines of code require more tests than others. Moreover this can easily by manipulated if the teams feels pressure to reach this level. A better approach is to determine all the scenario’s that need to be tested. This gives us confidence that these scenario’s do work, there might however be other case that have been forgotten or due to implementation details act different from others. Determining the scenario’s in a white box fashion will prevent this as the implementation details are used. But the cases that have to be tested will change often as implementation changes, and verifying that all cases are covered (and re-determining the interesting cases) is a manual action that will take up a lot of time. A much better way is to automatically gather information about which lines of code are executed by running the tests, called code coverage.

Continue reading