In this post we continue the work started with my colleague Philip Robinson in Belfast which was introduced in a previous post. The post motivated on "Effective Cloud-based Software Testing" as an important topic for research on testing cloud-based applications due to their complexity as well as for lowering down costs of testing operations. A subset of criteria from a paper published last year were also introduced as a mean to select a cloud testing pattern.
Here we argue that not only the cloud testing pattern is important, but also the cloud provider itself, since each has different hardware, configuration, performance, etc.. In fact different cloud providers under the same cloud testing pattern configuration will give different results. Although big cloud providers are already established in the market, other cloud providers are quickly proliferating. Also, new providers as well as "older" ones continuously introduce new features and improve existing ones. This reinforce the importance of selecting the right cloud-based software testing provider for effective testing.
Over time with experience we have reviewed the whole set of criteria as follows:
- Simplicity
- Observability
- Controllability
- Repeatability
- Predictability
- Target representation
- Cost effectiveness
- Reliability
- Availability
Reliability and availability have been added to the set as they have been proved to be of fundamental importance in cloud environments. The next step in this process is to introduce quantitative measures of these criteria so that an automatic testing framework can be developed. This should allow evaluating a cloud provider effectiveness in the testing process of the System under Test (SuT). However, performing this process to multiple cloud providers sequentially will be time consuming and in the long term overall not really cost-effective. We therefore envision a higher level testing framework which allows to target multiple cloud providers simultaneously to perform the application tests, compare the measured criteria and derive the most effective one according to the tester requirements.
Any combination of cloud testing patterns and cloud providers should be possible. For example different cloud testing patterns may be configured over the same cloud provider. The idea is that a tester can deploy simultaneously multiple instances of the same SuT and TS over multiple providers with whatever desired testing pattern and gather results so that comparisons over the above criteria can be performed.