Don’t monitor the health of your test cases, monitor the health of your product requirements!

[DirkLehmann]

[tl;dr]

Modern software engineering practices require a high-quality mindset. Automated tests such as unit-, integration-, system- or any other tests such as security or performance tests are on the rise. But with more and more automated tests, teams face a complex challenge to understand which automated tests are related to which requirement. SAP recently open sourced the Continuous Traceability Monitor (CTM) which addresses exactly this problem.  

 

Forget-me-not

Following the shift-left quality approach, test criteria should be considered as early as possible in the software development lifecycle. In best case, the product backlog item (PBI) already defines clear measures how the new requirement is tested. Implicitly, this could be accomplished by consequent usage of Behavior-Driven Development (BDD), which enables automated testing of an PBI. Explicitly, this could be done by insisting on test criteria in the definition of ready.

Developers implementing PBIs are urged to write appropriate automated tests that safeguard the new requirement. After the release of the implementation, the relation between the requirement and the automated tests is endangered. As time goes by, the link between the requirement and the set of automated tests covering it, could be forgotten. Some of the tests might cover additional requirements over time and in best case it’s just very hard to recall the information, which automated tests relate to a specific requirement. Worst case, the team loses the relation.

Lacking this relation causes troubles in many ways: In case of failing automated tests, it is impossible to tell which product requirements are affected. Monitoring product quality solely on the level of test cases tends to fade-out the original requirements that those tests cases should cover.
Code refactoring will also get tougher as it’s more difficult to understand refactoring implications.
Keeping the link between a requirement, its automated test cases and their current test results enables the team to make clear product quality statements at any time.

 

The compliance case

The need to understand the relation between a requirement and its current quality is neither new, nor unique to software engineering. In ISO9001 chapter 8.5.2 “Identification and traceability“ talks about “[…] status of outputs with respect to […] requirements…” which could be translated as: You should keep track (->status) of our product (->output) requirements

At SAP it is mandatory for every product during a release to prove that all existing as well as new requirements are covered by tests and that these tests were executed successfully. In that regards SAP teams are used to the term “Traceability” which traces a requirement to test cases to test results.

 

Bringing it all together

Product requirements (or PBIs) are stored in project tracking systems (e.g. Atlassian Jira), test code is kept in a source code versioning system (e.g. git) and test results are generated in CI/CD systems (e.g. Jenkins) which makes it cumbersome to keep these three entities linked.

To help teams keep track of the relation “Requirement” <-> “Automated Test Case” <-> "Test result", SAP open sourced a tool called Continuous Traceability Monitor (CTM). CTM checks test results from your Continuous Delivery pipeline against its linked product requirements and informs you in case any of your requirements is affected by failing tests.
Don’t monitor the health of your test cases, monitor the health of your product requirements!

For further information on how to use CTM in your project, have at look at the CTM Getting started wiki.

 

About the author

Dirk is a DevOps evangelist and Continuous Delivery expert at SAP.

Since 2001 Dirk worked in various roles at SAP including Development and Operations.
Together with his former team “TwoGo by SAP” he established the first Continuous Delivery implementation at SAP, delivering value daily to their customers.

In his current role, Dirk is a Cloud Quality Coach at SAP with focus on Continuous Delivery and DevOps practices.

Dirk is a frequent speaker at international conferences on DevOps and Continuous Delivery and he's a co-organizer of DevOpsDays Zurich.

Contact:
Twitter: @doergn
LinkedIn: https://www.linkedin.com/in/dirk-lehmann-7712bb125/

 

We have taken all measures possible to make this post as accurate as possible but things sometimes fall through the cracks. In case you find any errors or inconsistencies please let us know.