Part of my job in the Chief Customer Office at SAP is to play detective.  One particular case involved a situation where the Systems Integrator said 90% of the test scripts passed but after a go-live date was missed it was learned that only 30% of the scripts passed.  The executive sponsor of the project told me he would have made different decisions if he would have know that there was so much more to test.  So who was right?  They both were…The problem was how the numbers were being reported.  The Systems Integrator was not including scripts that had been deferred to future testing cycles where the Executive Sponsor thought everything was included.

The following are four ways to make sure you report your testing results accurately:

Testing Progress.JPGInclude all Test Scripts

Always report on all the test scripts in your status meetings.  During a project, test scripts go through different stages of when they are ready to be tested.  Just because it isn’t tested during the phase of the project you are in doesn’t mean it should be reported.

Categorize the Different States of Test Scripts

Keep it simple…”Failed”, “No Run” and “Passed”.  For the example above, a additional category of “Deferred” would help give an extra level of detail.

Clearly Define The Categories

I define “Failed” as any step in the script didn’t work…No matter how major or minor.  Others would give a script a “Passed” if there was a work around the tester could do to get the script to complete.  Regardless, there should be a clear criteria and it should be communicated regularly to the Steering Committee.

Use a Colored Stacked Bar Graph

A good example, broken down by project area, is included in the graphic.  I love a good picture.  A stacked bar is easy to tell proportionally how much is in each category.  Also include the number of scripts in each category so the raw numbers are clear.  In this example, you can see that there are still 61 scripts that need to be tested in the PP area.

With these simple techniques, you will avoid confusion, have a much smoother project and not need the “Script Detective”.

Let me know your experiences below or e-mail me @

To report this post you need to login first.


You must be Logged on to comment or reply to a post.

    1. Mark Marcus Post author

      Hi Derek,

      In my experience, the right person to own this is the Project Manager.  They should gather the information, prepare the graph and present in the Steering Committee Meetings.


  1. Sergio Ferrari

    Do you have any clue in relation with Test Workbench offered by SOLMAN ?

    I mean, can you describe how to mark a test case as deferred ?

    Is the nice proposed chart generated by the Test Workbench directly ?

    thanks for sharing


    1. Mark Marcus Post author

      Hi Sergio,

      I have not used SolMan to to gather or report the testing statistics but is sounds like something I should look into.

      I have used tools like SharePoint to keep the repository/status of scripts and use Excel to produce the report.  Probably a more “brute force” method.



Leave a Reply