ptailor of SAP provided this webcast to ASUG this week. He wrote about how SAP is taking proactive measures to deliver product quality software here.
He explained that SAP creates a global quality plan, with one quality lead from each product area. SAP has domain experts. SAP started this with BI4 FP3. Pritesh is the quality lead for BI4.1. These are my notes that I took as I listened to the webcast.
Agenda:
- Testing Focus and Quality Levels
- Quality Levels Illustrated
- End to End Tests
- BI4 FP3 and beyond Test Approach
- Top Quality Strategy Enhancements
- Upcoming Quality Strategy Investments
- Question & Answer
Testing Focus and Quality Levels
Figure 1: Source: SAP
Figure 1 provides the testing types and focus. “Smoke” type testing includes logon and logoff. SAP has a build master with a virtual team, release management and review critical matters.
Build Acceptance is automated with high level workflows not experience issues, focusing on first outer experience with 50 tests per product
A “malbug” is a defect must be fixed in 24 hours, and it is tracked closely.
Pritesh said they have a “Quality first” mentality.
Area Acceptance includes core testing, with no high impact failure. Each product area works separately where teams can only push their code lines if testing is successful. The product has to remain stable and not impact other teams. They target that testing is 100% automated.
With defect verification includes when the defect is fixed, determine risk and impact assessment and determine if new regressions. This is used to decide what best course of action. They use a defect severity matrix to determine priority of defects.
Feature verification includes a focus on new features to test.
With validation, it includes complex data. They include several types of testing so can finish in a sprint
Legacy regression testing includes a focus on end user functionality so users can upgrade “without disruption”.
With System Validation they execute testing with products as a whole and product interaction, 3rd party platform support and language testing – usually executed manually
Quality Levels Illustrated
Figure 2: Source: SAP
Figure 2 was the result of a build of slides, starting with planned tests all the way through final acceptance.
End to End Tests
Figure 3: Source: SAP
Figure 3 shows end to end tests. These are not all the tests, according to Pritesh. Enterprise deployment scenarios include complex customer scenarios – various SSO, methods of sign-on and includes a complex environment set up. It often includes 20 different configurations, frequency and scope depends on what feel needs to be changed.
With Language Testing, they ensure product is translated correctly - special tagging, and the UI is corrected given the translation string – language specialist will review the strings.
Performance & Reliability includes workflow has acceptable performance and reliable testing, version to version and build to build comparison. The key is to test workflows don’t slow user. The multi-user testing includes running without error with 24-72 hour reliability load testing, simulating a real world situation.
Platform Support includes any given release and each product team will determine impact. Higher risk platforms are global.
BI4 FP3 and beyond Test Approach
Figure 4: Source: SAP
With BI4 FP3 SAP shifted to agile and quality approach. This is further discussed in Pritesh's blog above.
Release cycles are in sprints, sprints are in waves.
They have one "hardening" wave to finalize.
New features – any new feature – must include a feature verification quality level – feature validation quality level before wave completes. It can’t span multiple waves.
The purple includes high quality, where they share with internal / external (customer validation)
Within each wave they run legacy regressions.
On the left of Figure 4 is recurring where build acceptances are executed daily
Global acceptance goal is to meet quality goals. At the last week of each sprint they lock the code line and shift to global acceptance mode. Any regressions found must be fixed before moving to next sprint.
Pritesh defines quality targets – include specific KPIs around performance and capabilities; these targets are agreed to by the areas.
Top Quality Strategy Enhancements
Figure 5: Source: SAP
The upper right of Figure 5 shows the “Continuous improvement life cycle” where they identify areas to improve, prioritize – go through root-cause, and execute.
Throughout various stages of life cycle they have the “drive to improve”
Partner testing is where SAP invites partners onsite to validate upcoming features –upgrade, migrations, SDK, etc. He said feedback from partners has been great and SAP analyzes feedback on how to incorporate into product.
The Customer Scenario testing team is in Bangalore. This team acquires key customer data and workflow and tests them. They also do the following:
- Execute customer tests
- Not introduce customer regressions
- Looking for more data and testers
When there is a customer escalation, they look at why was missed in testing.
Defect thresholds are defined as a calculated # of defects fixed in hardening week – if surpassed and team pulls the line to stop new feature development to address defects. They ensure no huge backlog at the end, with a “Quality first mentality”.
Patch to Patch Forward Fitting is new in BI4 SP6 where all fixes from patches of older SP’s are added. SAP has received good feedback on this.
Upcoming Quality Strategy Investments
Figure 6: Source SAP
Figure 6 shows enhancements for continuous integration where tests are automatically executed.
With Code Coverage Analysis, the goal to see what source code is tested. They are starting a proof of concept with 4.1 release in BI platform
With Defect Projection Models, they are looking at defect trends, put into HANA to summarize to suggest which areas for testing focus – roll out this year and next year
Question & Answer (a subset):
Q: When start the new forward fitting?
A: BI4 SP6 start the patch to patch forward fitting process – fixes from previous get included
Q: Samples provided do not stay current between versions. Is there a clean way to uninstall/roll back to remove a SP?
A: Install team is taking a look; create backup of CMS and FRS before start install to help with rollback. Product owner needs to comment
Q: What is being done to improve quality of Information Design Team?
A: Doing root cause analysis on defects
Q: Is there a way to volunteer for testing?
A: You can be part of the customer validation program and provide feedback or partner testing to come onsite. For Customer scenario testing – contact Pritesh
Other:
There were several questions about the forward fitting strategy; christopher.vozella suggested attending this ASUG webcast on July 30: BI4 Maintenance Demystified
Additionally, on July 29 there is an ASUG webcast titled Exterminate Bugs Before you Upgrade: Building an Automated Regression Test Strategy with BI4 with SAP's james.rapp
- SAP Managed Tags:
