Skip to Content

In a world of rapidly changing business challenges, one thing remains a constant driver for success:  Quality.  At SAP, we strive to deliver the highest quality software solutions for customers ranging from small businesses to large enterprises across many industries and around the world. This commitment to quality extends throughout the SAP portfolio of products.  That’s why we reached out to our Business Intelligence customers and gathered feedback on previous releases of our Business Intelligence software.  Based on the feedback, we continually investigate methods which improve on our quality processes. Quality improvements have resulted from large-scale development methodology shifts all the way to individual product teams continuously improving their day-to-day processes. In this article, we will go over some of the improvements implemented on the SAP BusinessObjects Enterprise product line.

 

Switch to Lean/Agile Development

Starting from the BI 4.0 FP3 project, there needed to be a customer centric shift to our organization. As such, Lean/Agile development was adopted as the standard software development process. The diagram below provides a bird’s eye view of the quality cycles within a release. Note that in the scope of this article, we only discuss the high-level approach.

/wp-content/uploads/2013/03/timeline_197634.png

The release cycle is divided into monthly sprints, where three sprints are grouped into one wave. Incremental functionality, for a particular feature, is developed and tested within a sprint. At the wave boundary, the collection of incremental functionalities developed within the sprints constitutes to a feature which has been fully developed and fully tested. Depending on the scope of the release, multiple features are in development at any point in time.

Daily product acceptance tests are performed, using automation, to ensure that core workflows remain enabled for development to continue. At the end of each sprint, a suite-wide acceptance is executed to ensure that the projects quality targets are being met. Following the lean/agile concepts of “pulling the line” to maintain quality, new feature development cannot resume until the quality criteria has been achieved. This allows us to reduce the late stage effects of having a large scale effort to stabilize the product at the end of the release cycle.

As the release progresses, more emphasis is placed on stabilizing the product versus new feature development. A hardening wave is employed at the end of the release cycle to complete further tests such as Customer Focused Deployments, Languages, and Platform support. This wave is also used to resolve the remaining outstanding defects.

Quality Criteria

The following table provides a summarized view of the quality criteria achieved at each sprint, wave, and hardening wave. A detailed example can be found in the appendix.

Target At the end of each… Criteria (summarized)
Usable Sprint
  • Major workflows of new features are tested and function successfully
  • Core workflows of existing functionality contain no High impact failures
Potentially Shippable Wave
  • New features are fully tested (eg. Complex workflows, error cases, etc.)
  • Advanced and specialized workflows of existing functionality contain no High or Medium Impact Failures
Shippable

Hardening Wave

  • Product has been tested according to plan and is ready for release. Tests include Languages, Customer Deployments, Platform Support, Performance and Reliability
  • All outstanding defects have been resolved

Further Quality Enhancements

In addition to the shift to Lean/Agile development, the following are some of the quality enhancements that have already been implemented or are in the process of doing so.

Defect Thresholds. The concept of defect thresholds was introduced to prevent a large accumulation of outstanding defects. If a product team crosses the threshold during the development cycle, all new feature development is halted until the defect backlog is back to a manageable level for that team. Thus, quality is maintained throughout the development cycle and it minimizes the chances of defects being pushed out of the release.

BW Quality Task Force. Formation of a task force focusing on improving our BW Quality. The objectives focus on three main areas: Target consistent behavior across all the suite of BI clients; improve design-time and run-time performance of all BI clients with BW as a data source; lastly, improve general functional quality of BW integration.

Customer Scenario Testing. By evaluating key workflows and environments employed by our customers, we identify and close gaps in our test strategy. Also, a cross representation of complex customer environments (including data) are replicated in house. According to the test strategy of a release, a set of acceptance tests are executed on each environment to ensure no regressions have been introduced.

Integrate Customer Escalations into core testing workflows. Similarly to customer scenario testing, we review escalations raised by customers. Any gaps identified in our test strategy are supplemented by including further tests to minimize the possibility of re-introducing those escalations in further releases.

Partner Testing. We extend opportunities to our partners to access and test our upcoming releases. Partners are able to evaluate and learn more about various features via guided exercises or free form activities using their own supplied data. Partners provide us with feedback on the solutions which is then evaluated by the development team for inclusion in the release.

Going forward

As described, numerous changes have been made to our processes to better product quality. In the spirit of continuous improvement, we are further analyzing our processes via retrospectives held at various levels of the organization. New improvements are then prioritized and incorporated into upcoming releases.

Appendix

Example of Usable Quality Criteria:

Criteria Metric
Functional
  • Pass Build Acceptance Tests (BAT) on Windows and AIX
    • 0 NOW defects open
  • Pass Global Area Acceptance Tests on Windows and AIX
    • 0 Critical defects on existing features
Performance
  • Pass Performance-based Build Acceptance Tests (BAT) on Windows and AIX (90th percentile response times no worse than 5-10%)
    • 0 Now defects open
Reliability
  • Pass Reliability-based Build Acceptance Tests (BAT) on Windows and AIX (99% success rate for 8-24 hours)
    • 0 Now defects open
Product Standards
  • Functionality: 0 Compile Errors in JLIN Build
  • Security: 0 Fortify Failures n Area
User Stories
  • Implementation Team Sign-off on all New Features
Scrum
  • Demo of New Features to Product Owner and Stakeholders completed

* NOW Defect: Issues in Major workflows that would impact greater than 90% of customers. These defects must be fixed as quickly as possible.

To report this post you need to login first.

2 Comments

You must be Logged on to comment or reply to a post.

  1. Jason Cao

    Hi Pritesh, thank you very much for taking the time to share this with SCN members! Having worked as a product manager in a waterfall development environment, I can see the benefits of LEAN in the form of Agile development.

    Where do you see the greatest opportunity for Agile to make the biggest impact on quality?

    Regards,
    Jason

    (0) 
    1. Pritesh Tailor Post author

      Thanks for reading the article, Jason.

      Our strong focus on a high quality bar in our done criteria has had a large impact.

      For example, in order for a sprint to be declared complete, frequent acceptance tests are executed. Any major defects found must be corrected prior to sprint sign off. This decreases the defect processing time and prevents cascading effects that reduce the product quality.

      And in the spirit of continuous improvement, we have a number of quality enhancements in the pipeline. Not mentioned in my article, but we are improving our continuous integration
      systems and exploring new processes such as code coverage.

      I am excited about the quality improvements we have put in place since I joined as the Quality Program Accountable for the BI 4.x Suite. We continue to boost our quality processes and I’m confident that our customers are starting to experience the benefits.

      (0) 

Leave a Reply