The product we create is a combination of hardware and software, and recently we have introduced the policy of doing a factory acceptance test (FAT) for every vehicle (AGV) that leaves our company. The FAT is meant to make sure that every vehicle that leaves the company works correctly, at least that is the theory because in practice we still have a lot of problems with getting everything working on the customer’s site. The goal of this blog post is to go into depth about the reasons for doing a FAT, the pros and cons and why it is failing for us.
The idea of a FAT is good, as it prevents unexpected problems when at the customer. It is always a good idea to discover problems as soon as possible. To do a FAT test however, you do need the room to execute this. A FAT however is completely determined by what situations are tested and this is often limited by how well you can reproduce the situations you will encounter. This is currently our biggest problem, our FAT does not cover enough causing us to send an AGV to the customer as soon as it could do a couple of moves.
Moreover, our FAT is currently focused a lot on testing the software, but this doesn’t make much sense because the software will behave the same on different vehicles. Software should also be tested automatically instead of with each AGV. Of course testing it with an AGV is still a good test, but it should not happen every time if nothing has changed. What you should test with the FAT is the integration between software and hardware and the actual hardware itself. While the first is still covered, the latter is just completely skipped.
I spent some time on-site last weeks and we were exactly hitting these hardware problems. We had a bracket that was bent causing the fork height to be measure incorrectly in some cases, some sensors were badly positioned, nothing was decently calibrated and more of this. All of this was missed by the FAT because the only thing we focused on was checking whether the AGV drives and can do some picks/drops.
The focus on software (and software/hardware integration) is even more ridiculous if you know that the first thing we need to do once we go to the customer is to completely wipe the AGV. This is a requirement because we have a write filter on the system that prevent corruption, it however also prevents us from changing the IP address. Since the IP address at our office is never the same as at the customer, this needs to be done. While this is something I feel needs to be changed as it also takes too much time to prepare this, at the moment this is just how we work.
To conclude I would encourage the use of a FAT, as the idea behind it is good. But expect to have a learning curve and adjust what is tested during the FAT to get the best outcome. Just like everything else, the results don’t come free. You need to put effort into the FAT, both the actual execution as the improvement. Think about what needs to be tested, and update it when you missed something to prevent it from happening again.