Abstract
SAP testing or ERP testing for that matter is complex. Tools are expected to simplify, ease the job. Market has so many testing tools licensed and freeware. Choices are so many that it is more often confusing than convincing as which tool to use. When and what tool to pick?
By the time a program gets to this question, there is already an existing framework which has to be embraced along. Framework might have to be changed to scope in tool. An in-depth analysis of tools purpose, tool’s integration with other tools, tool’s fitments to test framework and overall test strategy, ease of Tools customization capability should all be studied. All of this requires significant time and effort. It is critical to pick the right tool to enable program success. When done well, framework and tool becomes program enablers, making IT work efficiently and effectively release over release.
1. Introduction
Sooner or later every IT organization ends with bulk of code and tests around them. ERP testing being complex, has lot of artifacts making it a challenge to identify and run the right test set for a release. Given the huge number of test artifacts (like requirements, test cases, and use cases) and the complexity of AUT (Application Under Test), it has become a mandate to pick the right tool for managing effective testing. With software testing’s growing emphasis in past couple of decades; the market has a number of licensed and open source tools.
Automating the process of testing needs to be backed by a strong framework and right tools as enabler for process. Evaluating different tools based on need of the hour, keeping in mind the scalability and IT business growth is a significant activity. With so many options in the market how does one figure out as which tool to pick? Is there an all-in-one? Or does one have to go for different tools? In which case do these tools talk to each other? Are they integrated? Does the organization have to spend money and effort on all these tools at all times? Remember that it is not just the license, but the tool and technology savvy resource which adds to the cost. Number of organizations have sun set processes and retired systems as a price of incorrect automation. Right test/process automation is the need of the hour. With programs running on agile development methodology, it is even more challenging and compelling to have the right tools.
2. Identify ‘right’ tool
Every business challenge is unique. There is no one magic tool. Hence identifying the right tool as per the business challenge is critical. This tool decision is going to shape the programs short-term and long-term success. There are some basic yard stick questions that one has to evaluate the tool against to understand the tools success for the program.
- What is the Business case or tool need?
- Technology
- Tools Comparison
- Skill set availability in Market
- Cost – License, maintenance
2.1 Business case or tool need
In the SAP world, there are many tools. For instance, PANAYA, SOLMAN, ALM, QTP… When considering automating a process, which is the tool to use? All these tools work on SAP. But Why QTP or why not PANAYA?
Each of these tools enables/serves different purpose. So they are strong in those aspects.
For instance, ALM is a total release management tool. Hence this tool has better ways to organize requirements/BPT and map it to release and results. While Panaya is a testing tool which does record and replay or advanced scripting. So, Panaya will be more of a testing tool while ALM can be looked at an even widen horizon of release management.
QTP is again a functional testing tool. So should one go for QTP or PANAYA? This boils down to business need.
What is the objective of using the tool?
- Is the business trying to achieve better program/release management through automated testing? In such a case, a functional testing tool like QTP can be considered which has ease of integration with test management tool ALM. Although PANAYA is integrated with ALM, QTP better talks with ALM.
- Is the Business trying to cut upgrade cost? Is the overall objective to quickly understand test set release over release? In this case, PANAYA would be the choice to go for.
It is important to make today’s decision keeping in mind the future. Questions about scope for expansion should be thought for. For instance, is ALM used currently, or thought of in the next phase of the program?
2.2 Technology
Almost most of the testing tools available in the market these days are based on the concept of object oriented programming. They read an object of the AUT read or interpret its object property values and reproduce actions performed on them. So effectively this becomes another level programming over the existing AUT. Each tool uses a different technology to accomplish this task and hence uses a scripting language to achieve this task. For instance QTP and PANAYA use VB script, while QFTest uses Jython.
Technology also attributes to the scalability of the program. As program scope increases release over release, it becomes inevitable that these tools have to be integrated for business needs.
For example, as the build is built, tests have to be fired; automation tests to be completed and results to be available in ALM. All these jobs happening unattended overnight. Such scenarios are the truly automated ones without manual intervention and solve the purpose of testing as soon as the build is made. While building such a scenario, different tools have to be made to talk with each other to achieve this purpose.
ALM and QTP are products of the same company. So they speak to each other better. Getting product support for such a scenario is simpler. Hence one has to factor in the technology aspect in terms of cost and scalability while making tools selection.
2.3 Tools Comparison
Given the vast choice of tools, it is wise to follow a systematic scientific approach to picking a tool. For instance pugh matrix is one six sigma technique which can be used to narrow down and understand tools capabilities. Here is a pugh matrix comparison of three tools ALM, SOLMAN and PANAYA.
Rational used to get to the matrix is as below.
Requirements to Test
On Release management ALM, has the best features. From Release management to requirements tab, from requirements to BPT, HP ALM has all these features. So it scores over the other tools in comparison. Panaya integrates with ALM to leverage its advantages.
Test Management
ALM has advanced test management features. SOLMAN has good test management features. Usability and versatility is low though. Panaya leverages ALM for test management.
Test Plan
ALM has wide range of test plan features – from test creation, re-usability, rebuilding over existing tests, ALM has it all. SOLMAN caters all requirements of test management. PANAYA has its tests plan which can be used along with ALM to make the best use of PANAYA.
Test Package/ Test Suite
ALM and PANAYA have this feature which SOLMAN does not have this capability. Extensive manual and automation test can be performed with ALM. PANAYA can also do manual and automation test. The manual test execution of PANAYA is challenging though. Mapping “Test Package/ Test Suite” as a key tool feature against our need for picking a tool for program, thought process has to be aligned with overall business objective to decide as which tool to pick.
Test Result Management
ALM has standard test reports. Customization of these reports is possible. Quires can be created to create totally new test results. ALM thus offers from standard to build-your-own capabilities. PANAYA has a standard test report. It leverages ALM to take this to next level.
Create/Maintain automated Test suite
Panaya is a testing tool built for this purpose. Hence has the best of these features. For instance, object identification is a basic step which PANAYA does to create a basic record and reply of test. On the other hand, ALM or SOLMAN does not have a feature like this and one has to write complex programs to achieve a verification/validation.
Impact Analysis
Impact analysis is an activity done at different stages of a project/program execution.
For instance, impact analysis can be done after a defect fix to identify root cause. At those times requirements of the tool are different. Similarly impact analysis can be done at requirements level for a release to identify scope of testing. At that time tool capability needs are different. Post to a ‘go-live’ an impact analysis can be done to understand release quality. At this time, tool capability expectation is different. All these tools provide capabilities for purpose for which they were built.
To summarize the pugh matrix helps us list strength and weakness of each of these tools, and in a comparative view, help analyze tool which best suits the business/process need.
2.4 Skillset availability in Market
Good automation test resources are in demand and scarce. While evaluating tools it is significant to pick a tool whose technology is common, simple and widely used. For this will attribute to the maintenance cost. The rarer and specialized technology/scripting language, more the cost.
Resource skillset will largely influence to the capabilities a program can scale up to. For instance, getting custom feature implemented between ALM to QTP can be done with a moderately skilled resource. While custom feature between ALM and PANAYA requires specialization and expertise.
2.5 Cost – License, Maintenance
The ultimate objective of automation (test or process) is cost reduction. Reaping the benefits of cost ROI in few release cycles is the end objective. Hence Cost should be a driving factor while considering a tool. It is worth to mention that cost not only means cost of License but also skilled resource cost.
3. In Summary
Making selection of the ‘right’ tool is a critical task. It takes expertise, effort and time. For long successful project execution, efforts have to be streamlined with a scientific approach to get desired result. There is no one right tool. Needs, business priority and options have together be evaluated. A right choice at the right time creates history.
4. About the Author
Gayathri Palanichamy, Manager - Projects, IGATE
Gayathri is a PM with rich technical experience in automation testing and test framework. She is a champion in Agile test management and various testing tools. She has worked on different testing tools like ALM, QTP, Winrunner, QFTest, Panaya and has hands on experience in tools integration. Strongly backed with her work experience in Oracle as a SME for Oracle PLM, she has managed teams working on different products of SAP and Oracle.
Glossary
AUT – Application Under Test
- SAP Managed Tags:
