With a proliferation of (smart) mobile devices, today’s work place is allowing more and more people to bring their own devices to work (Bring Your Own Device). This naturally means companies have to manage a multitude of devices leading to what is now known as Mobile Device Management. However, it does not encompass everything there is about managing mobile devices. After all, device management at end of the day is mostly about procuring and managing the device. Enterprises need to look beyond the hardware aspect. In her blog SAP’s Milja Gillespie is calling this Enterprise Mobility Management. Milja notes that Enterprise mobility management is about the following-
- Mobile Inventory/Asset Management
- Mobile Application Management
- Mobile Security
- Wireless Expense Management
- Mobile Operations Management
- Mobile Help Desk
I envision that within Mobile Application Management, Testing and Validation will play a bigger role than it is currently. Companies are aware of the seismic shift that is happening in mobility arena but are not prepared for it to the full extent. According to current research by year 2012, companies will have, by a factor of 4 to 1 more mobility projects than the desktop variety. I think there is a gap when it comes to testing applications for mobile (regardless of the technical platform) – think about it this way-- conventional testing methodologies rely upon harnessing the Graphic User Interface (GUI) for traditional applications (Windows, Mainframe, Portals) to “validate” the delta between desired and actual outcome. With the mobile devices what we have instead is Natural User Interface (NUI). If you have used Xbox Kinect (it uses spatial gestures instead of a hardware device to interact with the system), you would have a fairly good example of what NUI is. You swipe, You pinch and zoom, you tap, you get the idea.
Given the revolutionary way we interact with our mobile devices, any strategy we adopt to test the applications for mobile must also take into account these differences. Given below are some challenges that will have to be overcome when instituting a test harness for mobile applications-
1. Device (or lack thereof) itself- start from the most basic premise. To test an app, you need the device it will run on. It is the norm rather than exception to roll out an app for most major mobile platforms. One my favorite apps is Pulse. And I can access it on my iPad as well as Android phone. Then there are hardware manufacturers. As of writing this blog , there are at least a dozen hardware vendors that are dominant in global mobile device landscape. These include heavyweights such as Apple, Nokia, Motorola, Samsung to name a few. Make sure you have access to most if not all configurations to test your app on.
2. Automation – When I am writing a program to automate a traditional desktop application test script, I can count on standard tools such as Quick Test Pro (QTP) and Load runner. These work on object recognition principles and often use APIs that are freely available. However, in the mobile device world we are stuck with no or very little object recognition information (that is made freely available at least) and manual testing is sometimes the only way to test. Manual testing as we know can be quite expensive and time consuming. You also lose the advantage of running multiple iterations of tests under variety of conditions when you do not have automation.
3. Data related issues- most devices are built to enable the user consume a lot of data not produce it. In the application testing world this means considerable work having to generate this data to test the app manually. Your strategy must define a data management program.
4. Process maturity lacking – last but not the least, you have to work with limited processes and documentation. There is a dearth of knowledge and capability about testing mobile apps. Most QA organizations I have worked with have little or no process set up on how they plan to test their rapidly expanding mobile infrastructure.
What I just described represents challenges when creating a test plan or strategy for a Device Under Test (DUT). What comes next? Now that you have the BIG picture, the strategy taken care of, now is the time to focus on operational and tactical aspects of your test plan. Any test plan that aims to verify the integrity of a mobile device application should have a repository of test cases, design steps and scenarios that will rigorously put the application to work in below mentioned areas.
- Installation, Start up time – This should be a set of tests that ensure the app can be installed over the air (OTA). If the application is too large, it should notify the user that excessive data consumption could be possible leading to surprises in monthly mobile bill.
- Memory used – Memory consumption by the app should be measured. Ensure that memory is not used excessively, and that it does not lead to corrupt memory locations.
- User interface – This should be an array of tests to validate several aspects of user interaction with the interface. Ease of use, readability, consistency are some of the flavors you would include ion your test plan.
- Connections – Test to make sure app can function under variety of connectivity situations. It can handle broken connections, roaming etc.
- Performance – Resource consumption. Execute test cases to determine load factor and other metrics that ensure CPU utilization is minimal.
- Functionality – ensure that the app does not interfere with calls and other media messages.
- Media – test how the app functions with other settings tweaked. For instance, if you app is a media player, make sure it stops playing media when a headphone is ejected from the device.
- Data related – Run tests to make sure app handles data in consistent and safe manner. Check for insertion of new records, data deletion etc. Ensure you can clear temporary files and logs when indicated by the user.
- Security – Test for encryption, passwords. Make sure your app does not expose the device to security vulnerabilities.
I will conclude by noting that this field is in its infancy and there are miles to go before application testing for mobile reaches same maturity levels as testing processes for other mainstream platforms. There are new and exciting initiatives in this field and I have explored various vendors making testing software for mobile applications testing- many of these tools are building on VBA model or leveraging existing solutions such as QTP. It would be very interesting to hear readers thoughts and experiences how their companies are meeting requirements for mobile application testing.
