Requirement Coverage in Unit Testing

[vidya_gugnani]

All software designed / implemented adheres to some requirement either internal or customer-driven. The coverage of requirements is a fundamental need throughout the software life cycle. Be it during design, coding or testing, the ability to ensure that the software meets the expected requirements is something that every developer/project manager aspires for.  Moreover, as the software matures and goes into several iterations of enhancements and bug fixes, it becomes even more necessary to ensure requirement coverage in the software.   Several techniques have been adopted to ensure that requirements are met such as:      Use cases      Test cases       Design Documents      Spreadsheets with traceability matrix to requirements.  The main problem that then arises is that not all developers/managers use the same tool and hence yardsticks for measurement of the requirement coverage remain arbitrary and untraced.  A):How can Unit Testing be used for Requirement Coverage?           The purpose of software is to meet the expected requirements.  The intent of Unit Testing is to make the software more robust.   Why should these be integrated?  Though requirements and unit testing can be viewed in isolation, a combination of the two gives a more requirement focused development approach.   Unit tests need not be restricted to testing code for mere functionality or correctness or to resolve bugs.  Unit tests cannot guarantee that all requirements have been met and so neither can developers/managers.  Ultimately, the quality assurance team could catch a potential miss or a malfunctioning feature which could have been thoroughly tested.  Integrating unit tests with software requirements makes software more requirement robust and requirement focused.  With the integration of unit tests with software requirement, you also know exactly what requirements are missing.   B):Integration Unit Testing with Requirement Coverage in ABAP   Currently the ABAP Workbench offers developers the opportunity to write unit tests using ‘ABAP Unit’, a flavor of Junit in the ABAP world.  Developers can write tests and view results (pass/fail) to check the correctness of their code. ABAP Unit also proposes to provide a developer a rich integration with coverage tools which solves the developer's problem of checking for the coverage of the test code.  However,ABAP Unit still relies on the efficiency of a developer to test all possible requirements .  New features provided in ABAP Unit make for the generation of test cases much simpler and ensure more coverage of functionality,but not necessarily the requirements.  The question that now arises is: 1) How do we make sure that the code meets the requirements and to what extent?   2) With unit tests how can a tool help achieve measure this?  3) What should such a tool provide?  Let’s begin with the last question. A tool which provides this integration would ideally require to:  1.  Create or gather requirements. This would mostly be already done in most projects using an internal tool or otherwise. All projects would of course have their own Project ID and the list of requirements present in the pre-requisite documentation. The tool in this case would then just need to gather these from an existing document or gather it from the user who enters this input manually.  2.  Associate the requirements with tasks/projects.  This would be a useful feature when projects need to tracked to the minute detail of the requirement and when their associated unit tests need to be retrieved. However, for projects which do not need such a fine granularity, this feature could be an overkill and just a simple Project ID or any random id assigned by the user should suffice. The tool would then have to provide a mechanism to map a requirement in a task/project/sub-project to a unit test.  3.  Now comes the tricky part for the tester.  As you write unit tests, the tool would have to update the requirements file(s) to map the unit tests to the requirements.   Here now are the options available to us: 1)The tool captures all unit tests written and just matches them to existing requirements or  2)Test cases could automatically be created by the tool for each feature requested and a skeleton code resides which the developer could choose to use. T  This would have the additional benefit of also displaying clearly how many of the requirements have not been tested or worse yet, the software/code has not met the requirement or this has been omitted by the developer.  For this purpose, the tool could be integrated with already existing tools which provide the test-code generation aspect or capturing the requirements aspect.(e.g Abap Unit or Junit tests in Eclipse).  3)The best option of course would be that the tool could do both.  4)The tool would also have to consider if refractoring of the test-cases would need to be done once the requirements change and the old/redundant test cases be deleted or moved to a regression library.  5)Once the unit tests are run, the tool would ideally provide requirement coverage statistics for tests run with statistics indicating to what extent unit tests covered the requirements. This tool would then comprehensively help the code meet the requirements and a developer his testing needs.   C) Benefits of Requirement Coverage Based Testing:  Developer: 1)     Ensures code meets requirements. 2)     Ensures requirements are adequately tested. 3)     Requirement focused approach to coding and testing. 4)     Statistics available to management and the Quality team for testing statistics. 5)     Integrate with tools already used in testing/coverage analysis. 6)     Bugs identified earlier, savings in time and cost of the product delivery.  Management: 1)     Traceability of code to requirements and the unit tests. 2)     Increased confidence in the overall product quality. 3)     Plan for risk elements in the project better.  Quality Assurance Teams: 1)     Quality Assurance can use statistics from such a tool now to get a better picture of untested requirements or untested code or requirements not met to help catch bugs better.   D) A working sample of this concept:  JRequire is an existing tool which currently offers a similar functionality. Its integration with Junit and Ant provides us a model to base our tools on . A Junit flavor, ABAP Unit is already offered in the ABAP world for developers to write unit tests.  With JRequire, when you run unit tests (either in Eclipse or via ANT), JRequire can generate the requirement coverage report that gives you visibility into  •     requirements successfully covered by code: all associated unit tests pass  •     broken requirements: either associated unit test(s) failed or did not execute  •     requirements that are not covered: no unit test associated    As Jrequire is also available as an Eclipse plug-in,it offers developers working on SAP NetWeaver Developer Studio an opportuntity to implement requirement based testing easily in their Eclipse development environment.  Hopefully,usage and integration of such plugin in the test environment will ease and refine the testing process and bring us one step closer to that elusive software quality.   Useful Links for more information: http://zap-t.com/presentations/requirementdriventestautomationprocess.pdf  http://www.eclipseplugincentral.com/Web_Links+index-req-viewlink-cid-312.html  http://www.technobuff.net/webapp/product/showProduct.do?name=jrequire  http://www.junit.org