In a recent study, IBM reported that the average enterprise devotes up to 50 percent of its entire technology infrastructure to development and testing, but typically up to 90 percent of the test infrastructure remains idle. Whether or not this describes your company, the fact remains that test automation has increased both the size and number of test labs, where a test lab is a room or space dedicated to computers used exclusively for testing. With the advent of virtualization, computers in a test lab have been made far more productive along with the people using them – just the ability to store different configurations for use in testing has been a great boon for testers and developers everywhere.
Cloud computing, of course, is really just a specialized way of using virtualization, but it relieves you of the need to purchase, store and maintain your own hardware and provides virtually unlimited capacity for when you need it. Most organizations that use VMware or Virtual PC are at least studying the potential benefits of the cloud and many are making the move, either in whole or in part, for all the right reasons. What those reasons are and how to go about making the switch are the topics of this article.
Types of Cloud Computing
It’s become as tricky to define the “cloud” as it was to define “artificial intelligence” in the 80’s (if you can remember that far back). Microsoft’s Azure, for example, provides just about everything but the Windows platform itself as a cloud-based service, while Amazon and Rackspace provide both the Windows and the Linux platforms as a service. For test automation purposes you will most often need the Windows platform as a service, since most functional test tools (barring a few exceptions) run on Windows. So for the purposes of this article we will confine our discussion to clouds that offer the Windows platform as a service, and thus are compatible with tools like HP Mercury QuickTest Pro, Micro Focus SilkTest, Smart Bear Test Complete, Worksoft Certify and so forth.
Test labs with 10 to 20 or even more machines in them are quite common, even though they are going the way of the dodo bird as of 2010. Because they consume space and power and are usually idle a great deal of the time, they are prime candidates for migration to the cloud.
Benefits of the Cloud for Testing
Aside from the space and power consumption, what makes the cloud better than using virtualization in house? Well, the most obvious example is load testing, in which many machines are needed to generate load. The precise number of machines is dictated by the size of the load that is necessary for the test, but I have seen test labs with 100 computers stacked from floor to ceiling dedicated entirely to load testing and even then it wasn’t enough.
Even for functional testing you may want to avail yourself of the added capacity of the cloud for a number of reasons. You might want to run a number of tests in parallel and thus cut down on the time needed to complete a lengthy test run. For cross browser testing you may want to run tests in all the major browsers instead of just one. Practically anything you can do with virtualization in house you can do in the cloud, but without the limits inherent in a lab of any size.
Choosing the Right Cloud
Everyone knows that Amazon was the first to introduce cloud computing to the masses and remains by far the most popular and well known cloud. The Amazon Compute Cloud (EC2) supports a wide range of Windows and Linux platforms in all sizes, and takes just minutes to configure and deploy custom test machine images. Rackspace has also been an innovator in this space and has recently announced support for Windows as well as Linux platforms. Pricing ranges from 2 cents an hour to over $1 an hour based on the size of the machine in terms of memory and disk space as well as the speed and number of CPU cores.
One of the benefits of the Amazon cloud is something called “Virtual Private Cloud”, which allows you to configure your test machine instances to run inside your firewall. Unless the application under test runs on the client machine itself you will often need to have your test machines run on the internal network, and VPC supports this. A workaround might be to install VPN software on your test machine images, but in any case it is a matter to consider.
The location of data centers is also a factor, and Amazon has a presence in the USA, Europe and Asia. This may often be less important for functional testing than for load testing, however, because load testing must consider key test result metrics such as network latency.
Some Drawbacks and Limitations
OK, so you’ve set up an account at the cloud provider of your choice and fired up an instance of a Windows or Linux machine. Now what?
Well, like any “vanilla” virtual machine you’ve got to copy whatever files you need, including programs to install. For this you need to rely on FTP commands, which is cumbersome compared to copying files from one drive to another. You will need to use RDP for Windows or SSH for Linux to log in, then attach to yet another server to download your files. With Amazon’s Virtual Private Cloud things are easier since you can “see” the machine from inside your firewall, but still you may find yourself doing a lot of manual work.
Another limiting factor, is the availability of Windows platforms – neither Amazon nor Rackspace supports anything but Windows Server editions, whereas VMware supports XP, Vista and Windows 7 as well. With Microsoft promising to provide Windows as a service on the Azure cloud, maybe they will lift this limitation.
Finally, a remote connection is never as fast as a local one so screens may not appear as quickly as you are accustomed to. And yet for the number of advantages you get these limitations are worth it for most people.
What about Security?
Security has always been a major concern, and even more so nowadays with so many existing cyber threats. While the data centers themselves are physically secure, what you expose to the outside world is entirely up to you. With Amazon, for instance, the use of “security groups” allows you to set the firewall options you require and attach them to your instances on startup. The default is “all ports closed”, so even to use RDP and SSH you will have to create a security group with these ports open.
When you spin up a cloud instance you are provided a public IP address that anyone can access. This IP address lives only for the duration of that instance’s lifetime so if you want to maintain that same address in the future Amazon provides “elastic IP addresses” which are static and allow you to associate them with any instance. If you need to make sure nobody outside your organization can access your test machines you should consider using VPC so that your instances run on a sub-net inside your private network.
Load Testing as “Killer App” for the Cloud
Hardly anything else comes close to load testing as a perfect reason to use the cloud. Depending on how big of a test you need to run you can easily require dozens if not hundreds of computers to generate the load. It simply is not practical for most companies to have this many computers on hand for testing purposes, especially if they are only needed from time to time. With cloud computing you can spin up as many as you need, use them for your tests, and tear them down again.
There is a running debate about the accuracy of load tests that use the cloud for load generation (or any virtual machines for that matter) because of the slowing down of the system clock that can occur if the cloud provider’s resources become overly taxed. It’s been our experience that this effect, while certainly a possibility, occurs rarely enough and is subtle enough to have no visible effect on the results. In any case, if you need 100 computers to run a load test what are your options other than using the cloud? It’s one of those trade-offs that is easy to make for most companies (unless you are testing medical or transportation equipment where milliseconds can mean the difference between life and death – fortunately most web apps are nowhere near as sensitive).
Making the Switch
In general, the procedure for setting up a test lab on the cloud is not much different than setting one up in a virtualized environment. Start with a “plain vanilla” system, install and configure whatever software you need, and save it. The magic, of course, is most visible when you need to set up and tear down your instances – watching a dozen or more test machines come to life in a matter of minutes is a fun exercise, particularly when they are churning out tests and test results.
Given some of the drawbacks and limitations that still exist with the cloud many companies are employing a hybrid approach that uses a combination of virtual machines and cloud machines. But the fact still remains that the days of dedicating a physical computer to run a single test are gone – have you noticed?