Advanced Test Orchestration using Test Plans in SAP Cloud ALM

In this blog post I would like to provide some details about using test plans for advanced test orchestration in SAP Cloud ALM. The blog will address the following questions:

1. When do you need test plans in SAP Cloud ALM?
2. How to use test plans in SAP Cloud ALM?

1. When do you need test plans in SAP Cloud ALM ?

In larger implementation or transformation projects you might want to run your test activities in iterations and so need setup multiple test cycles re-using the same or a similar set of test cases.

On top, you might want to roll-out the implemented solution or a global template to multiple organizations like subsidiaries, plants or markets which require dedicated tests per roll-out. Thus, you might want to re-use most of the test cases.

Although it was possible without test plans to run the same test case again while creating new test runs, the overall test status was always set to the status of the last test run.

With the following illustrations I would like to explain the underlying concept with the existing approach without test plans and the new approach while using test plans:

1.1. Lean Test Orchestration Without Test Plans

Each test case appears in the Test Execution app as soon the status is set to ‘Prepared’

When the execution is started, a test run is created, status can be set, and defects created

In the following example, the first test run was set to ‘Passed’ after all test actions were completed successfully and the test run was set to ‘Finished’.  Then a second test run was started and test action 1.1 was set to ‘Fail’. In consequence, the overall test status is displayed as ‘Failed’.

Test case with two test runs.

After introducing test plans, this lean approach is still available, so you could continue without any overhead. In most cases, this is sufficient for smaller projects and public cloud implementations with an agile or lean implementation approach. Therefore, we still support executing test cases without the need for test plans.

All prepared test cases have been completed.

1.2.    Advanced Test Orchestration with Test Plans

For more complex projects, customers asked for the possibility to have a dedicated execution context for each test cycle.

To achieve this, you could create a test plan and assign relevant test cases

A subset of test cases have been added to the test plan that is still ‘In Preparation’

As the test plan status is still ‘In Preparation’, no occurrences of the assigned test cases are displayed in the Test Execution app yet.

After switching the test plan status to ‘In Testing’, all assigned test cases in status ‘Prepared’ get a new occurrence (execution context) and the test execution can be started.

The test plan is now ‘In Testing’, so assigned and prepared test cases can be executed in their own execution context

Another test plan has been created and is ‘In Preparation’ with a similar set of test cases

The next picture shows the situation after Test Case 3 has been set to ‘Prepared’ and Test Plan B has been set to ‘In Testing’.

Setting the new test plan to ‘In Testing’ will show related test cases with a new occurrence in status ‘Initial’ to start a new test cycle. Test Case 3 is now available for execution in three contexts (Test Plan A, Test Plan B, none)

How to use test plans in SAP Cloud ALM?

New Test Plan App

There is a new tile in the launch pad called Test Plans:

New tile ‘Test Plans’ in the SAP Cloud ALM launch-pad

When you open the Test Plan app, you will see a list of existing test plans with their status, time frame, and the aggregated preparation status and execution status of assigned test cases:

Test Plan Overview

You can create new test plans choosing ‘Create’ to get this detail screen in draft mode:

Detail screen for new test plan

To assign test cases choose ‘Assign Test Cases’ to get this simple assignment dialog:

Value help for test case

You can either start typing in the field to get a prefiltered list of test cases, or you could open the value help to get an advanced assignment dialog with search and filter options:

Advanced assignment dialog with search and filter capabilities

After confirming the selection and the assignment dialog, you will come back to the test plan details in draft mode, showing the assigned test cases:

Test plan details in draft mode with assigned test cases

In draft mode, the test plan is stored but not yet available to work with. Only when you choose ‘Create’, there will be a ‘saved version’ that can be used for processing.

Whenever you open the test plan in edit mode, a draft version is created, which contains all your changes without saving. You can switch between the draft and saved version of your test plans next to the test plan title:

Draft handling for test plans

You could even close your browser without losing your changes.You can find more details on draft handling here.

Besides the test plan statuses ‘In Preparation’ and ‘In Testing’, there is a status called ‘Finished’, which indicates that the test phase / test cycle is finished. This status does not have any impact on the visibility of related test cases occurrences in the Test Execution app.

Enhancements in the Test Execution App

In the Test Execution app, you will find a new filter for test plans and the occurrences of test cases are grouped by test plans.

Without setting the filter for test plans, you will see at least one entry per prepared test case belonging to the group ‘(None)’ . In addition, you’ll get another entry for each test case per test plan that is in status ‘In Testing’ or ‘Finished’.

Test Execution app with multiple occurrences of the same test cases

When planning to work with test plans, you could simplify the entry for your testers by setting the filter for the right test plan(s) and then sharing the URL.

Enhancements in the Test Preparation App

In the Test Preparation app, we have also added the filter for test plans to be able to reduce the list of test cases of the whole project to test cases that are not assigned yet to any test plan, or to list all test cases already assigned to a specific test plan.

Test plan filter in the Test Preparation app

Outlook on Planned Enhancements*

We plan to enhance the test plan capabilities in several iterations. In one of the next iterations we plan* to provide the following enhancements:

Analytics:

Filter on test plans in following dashboards:

• Test Execution Analysis and Defects Reporting
• Solution Process Traceability
• Requirements Tracebility
• User Story Tracebility
• Test-related cards in Overview app

Test Case Assignment to Test Plans

Advanced test case selection

• with additional filters in the Test Plan app
• by levaraging the Traceability apps to select related test cases for assignment to a test plan

Tester Assignment:

Assign testers to test cases

• Default assignment without test plan
• Dedicated assignment per test plan

Copy of Test Plans

Test Sequences:

Define the sequences of test cases within a test plan

Roadmap for Further Planned Innovations:

You can find the latest updates on our planning in the Roadmap Explorer for SAP Cloud ALM  

* The information in this presentation is confidential and proprietary to SAP and may not be disclosed without the permission of SAP. Except for your obligation to protect confidential information, this presentation is not subject to your license agreement or any other service or subscription agreement with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or any related document, or to develop or release any functionality mentioned therein. This presentation, or any related document and SAP’s strategy and possible future developments, products and or platforms directions and functionality are all subject to change and may be changed by SAP at any time for any reason without notice. The information in this presentation is not a commitment, promise or legal obligation to deliver any material, code or functionality.  This presentation is provided without a warranty of any kind, either express or implied, including but not limited to, the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. This presentation is for informational purposes and may not be incorporated into a contract. SAP assumes no responsibility for errors or omissions in this presentation, except if such damages were caused by SAP’s intentional or gross negligence. All forward-looking statements are subject to various risks and uncertainties that could cause actual results to differ materially from expectations. Readers are cautioned not to place undue reliance on these forward-looking statements, which speak only as of their dates, and they should not be relied upon in making purchasing decisions.