Technical Articles
Manual and automated testing
In this blog-post i wanted to point out, why and when we need manual and automated testing. Also automated testing is getting more and more popular, manual testing has still some benefits, which are hard to achieve with automated testing.
The benefits of manual testing
Automated tests are build with determinism in mind. So they are mostly static without the random events and states, that occur in the real world. The dependencies are replaced by test-doubles, which don’t necessarly mock all the states and events, that occur in the real dependencies.
Let’s take an example. Recently i had to develop an application, that books consumption quantities, surcharges and production quantities related to a process order. After each booking step, the SAP-LUW needs to be closed due to the restrictions for booking material documents. If one step failed, the process could be repeated without booking the consumption or the surcharges twice. My solution was to set the withdrawn sign in the reservation and only book the consumption when this withdrawn sign wasn’t set. When writing the automated test, i created process orders and used the real dependencies (BAPI_GOODSMVT_CREATE) to book the consumption and the production. Every test-case showed green, but in production it fails. What was the reason? The process orders were confirmed before the consumption and the production was booked. The confirmation has set the withdrawn sign. So the consumption step was never booked in the production environment. Why we didn’t found that bug in the tests? Because the tests were static and we didn’t consider the events like the confirmation, that happened on the process order in the production environment.
Through random tests it’s possible to discover bugs, which slipped through the automated tests like in the example above. Manual testing brings new variations and helps to randomize the tests.
Why automated tests are still needed
Although determinism is some kind of drawback, when it comes to new variations, it’s a benefit on the other site. With determinism it’s possible to execute the test-cases whenever needed, but they hardly have any variation. So it’s very important to design the test-doubles and the database entities used for testing accurate, that they act like the real dependencies.
Automated testings aren't really about testing, in my view, they're about protecting your code base from changes that break it. If automated tests fail, you know there's an issue. If they pass, it doesn't mean that there isn't an issue somewhere.
If your tests cover all scenario within a class, then dependencies don't matter as such. If you have an issue, then you know the problem doesn't lie within that class.
Automated testing can never replace manual testing. It's a very useful adjunct though.
Where possible (and it isn't always) I write automated tests for issues that manual testing find.
This is proparly true for all kinds of tests. When the best test you can't guarantee the absence of issues and sometimes we can't discover them with deductive methods. Instead these issues are found by actively using the software. This is in my opinion one reason, why software, that you develop and use by yourself is often better than software, that you develop for someone else and don't use actively.
According to the Halting Problem, this is certainly true! You can never guarantee that code in a Turing Complete language is correct.