What Is Software Testing?

By Hans Hartmann

Software glitches have always caused major damage. And recently, the biggest failures haven’t been in aerospace or defense, but have increasingly affected the average consumer. Often, we hear that the software testing wasn’t done properly. But what does this mean exactly?

software testing is required with today's complex programs

 Photo: iStockphoto

Every now and then, really spectacular software breakdowns occur. The opening of Heathrow Terminal 5 became a public embarrassment because the baggage system failed to function. More than 17 million customer accounts at RBS and its subsidiaries NatWest and Ulster Bank could be accessed for some or all of the day because the installation of customer management software corrupted the entire system. One of the biggest Austrian banks paid out €21 million to appease its customers with vouchers because the new online banking software didn’t work for days on end.

Errors like these are not only damaging to a company’s brand, but can also be very costly. The goal of software testing is to avoid such incidents and the consequences. On the following pages, we explore the topic of software testing and address these main questions:

Hans Hartmann, test director at Objentis since 2007. (Photo: Private)

We can assume that in the cases previously mentioned, the software in question was definitely tested: Banks and insurance companies know the risks of using software that has not been tested. So how can such malfunctions continue to occur? Some, but certainly not all, software glitches can be caused by storms and natural disasters. Still, this provides no explanation for the increase in software errors of late. Testing has always been done and it used to work well. And natural disasters are a known, if unpredictable, factor. So why should the tried-and-true formulas suddenly fail?

The reason is simple: Programs have become more complex. And to address this complexity, more testing is required. How much more? Take the years 2000 and 2010. In this time, the volume of data being moved around increased by a factor of 50,000. If a program was tested for two weeks in 2000, it would have to be tested for 100,000 weeks in 2010 – in other words, around two thousand years.

More interactions, not more data, increases complexity

Working and calculating this way is clearly not an option. After all, software is now more efficient, development tools allow many errors to be detected before the program is first created, and modern object-oriented software design enables developers to code neatly and in a less error-prone way. But even if testing is only increased by a factor of 50, it would still have to be tested for 100 weeks – or two years. That simply isn’t feasible.

Comparing the difference in size and quantity alone doesn’t necessarily mean that the software has become more complex. In fact, one of the main arguments for using a computer is that it doesn’t matter whether it has to perform a calculation five times or 5,000 times. It should simply be reliable. It is not the increase in the quantity of data that causes complexity, but rather the increase in possible connections and systems.

Look at the development of mobile telephony: In Germany, Radio Telephone Network C came along first in cumbersome cases, followed by the much more manageable digital cellular network D-Netz. In comparison, today’s smartphones have the processing power of mainframe computers from 20 years ago. Apart from the pure advancement of technical data, think about all the things that can now be done with a smartphone. Above all, think about the number of other systems that can be tapped into – at the same time, even. It is the number of possible connections that causes the corresponding increase in complexity.

The main difference between today and yesterday is not the advancement in programming languages – even though developers may no longer code in Assembler or COBOL, these languages can still be used to write good programs today – but rather the number of possible solutions there are for a certain problem.

Take this analogy of trying to cross a river that is 30 feet wide without using a boat and without getting wet. In the past, there was one solution: system analysts would look for places where big rocks could be used jump across the river to the other side. Today, there are 10 different bridges crossing the river, that is, 10 different ways to solve the problem.

The software architect, then, has to choose a particular solution based on whether it meets various quality requirements. Let’s say there is a highway bridge crossing the river as well as a wooden walkway. To use the highway, you need to build feeder roads. Even if the simple wooden walkway is sufficient and building feeder roads requires more effort, the software architect may still choose to use the highway with the reasoning that other people want to cross the river, too.

It’s impossible to test every combination

Here is another example: Forty years ago, when passengers would buy a train ticket from a ticket machine, they would have to answer a series of questions, one after the other. From where do you wish to depart? To where do you wish to travel? How old are you? Are you entitled to a reduced fare? In which class do you wish to travel? And so on. If they discovered while answering the questions that they didn’t have enough money, they would have to cancel the transaction and start again from the beginning.

At today’s ticket machines, passengers will find the questions slightly more hidden in different fields. Instead of entering their age, they select standard fare, half price, or other offers. Rather than typing the destination in full, they type the first few letters, and only the possible destinations are then displayed. While the layout of the input fields suggests that the information can be entered in any order, that is still not possible. For example, if users have entered a discount ticket, they cannot subsequently upgrade to first class. However, instead of getting an error message that says, “First class must be entered before you select a discount,” users will see a message like, “You must purchase your first class ticket on the train.”

In this case, it is clear that developers made some small mistakes in the process of transferring an originally linear, simple input sequence to a graphical input system. Let’s say the machine needs to process five different inputs and they can be in any order. This means there are 120 different combinations of how entries can be made. So, it is understandable that not all input options were tested before the software was implemented.

In the past, it was possible to test each individual function and then test the complete process. Now it is necessary to test the interactions between individual functions. The number of these interactions depends directly on the number of possible sequence combinations, which can easily be a seven-digit sum. If you take a smartphone, for example, the number of possible combinations surpasses the example of the ticket machines by several orders of magnitude.