Skip to Content

Note: I did originally publish in the following post in my company’s blog on software quality. Since it might be interesting for developers in the SAP BW realm, I re-publish it here (slightly adopted).


As my colleague Fabian Streitel explained in another post, a combination of change detection and execution logging can substantially increase transparency regarding which recent changes of a software system have actually been covered by the testing process. I will not repeat all the details of the Test Gap Analysis approach here, but instead just summarize the core idea: Untested new or changed code is much more likely to contain bugs than other parts of a software system. Therefore it makes sense to use information about code changes and code execution during testing in order to identify those changed but untested areas.

Several times we heard from our customers that they like the idea, but they are not sure about its applicability in their specific project. In the majority of these cases the argument was that the project mainly deals with generated artifacts rather than code, ranging from Python snippets generated from UML-like models and stored in a proprietary database schema to SAP BW applications containing a variety of artifact types beyond ABAP code. Even under these circumstances Test Gap Analysis is a valuable tool and may provide insight into what would otherwise be hidden from the team. In the following I explain how we applied Test Gap Analysis in an SAP BW environment.

The Starting Point

As you all know, in SAP BW a large amount of development is performed graphically in the BW Workbench. In cases where custom behavior is required, routines can be attached at well-defined points. As a consequence, it is very hard to track changes in a BW application. Of course, there is metadata attached to every element containing the relevant information, but seeing all changes that have occurred since a given point in time (e.g., the current production release) is not a trivial task. The same holds for execution information. Since we were already using Test Gap Analysis for transactional ABAP systems, we reached out to a team developing in BW and showed them some results for their own custom ABAP code.

/wp-content/uploads/2015/04/only_manual_code_695282.png
Figure 1: Test Gaps in Manually Maintained ABAP code only

The picture shows all ABAP code manually maintained by the development team. Each rectangle having white borders corresponds to a package, and the smaller rectangles within correspond to processing blocks, i.e., methods, form routines, function modules, or the like. As explained in Fabian’s post, grey means the block was unchanged, while the colors denote changed blocks. Out of these, the green ones have been executed after the most recent change, while the orange ones have untested modifications with regard to the baseline, and the red ones are new and untested. Of course, tooltips are provided when hovering over a rectangle containing all the information necessary to identify which code block it represents – I just did not include it in the screenshot for confidentiality reasons.

Moving Forward

As expected, the feedback was that most of the development effort was not represented in the picture, since development mainly happened in Queries, Transformations and DTPs rather than plain ABAP code. Another insight, however, was that all these artifacts are transformed into executable ABAP code. Therefore, we analyzed the code generated from them, keeping track of the original objects’ names. The result was (of course) much larger.

/wp-content/uploads/2015/04/all_code_695283.png
Figure 2: Code generated from DTPs (left), Transformations (middle top), Queries (middle bottom), and manually maintained code (far right)

Including all the programs generated out of BW objects, the whole content of the first picture shrinks down to what you can see in the right column now, meaning that it only makes up a fraction of the analyzed code. Therefore, we have two main observations: First, ABAP programs generated from BW objects tend to get quite large and contain a lot of methods. Second, not every generated method is executed when the respective BW object is executed. In order to make the output more comprehensible, we decided to draw only one rectangle per BW object and mark it as changed (or executed) if at least one of the generated methods has been changed (or executed). This way, the granularity of the result is much closer to what the developer expects. In addition, we shrink the rectangles representing these generated programs by a configurable factor. Since the absolute size of these programs is not comparable to that of manually maintained code anyway, the scaling factor can be adjusted to achieve an easier to navigate visual representation.

/wp-content/uploads/2015/04/all_code_adjusted_695284.png
Figure 3: Aggregated and scaled view for generated code (left to middle) and manually maintained code (right)

The Result

With this visualization at hand, the teams can now directly see which parts of the application have changed since the last release in order to focus their test efforts and monitor the test coverage over time. This helps increasing transparency and provides timely feedback regarding the effectiveness of the test suite in terms of change coverage.

To report this post you need to login first.

3 Comments

You must be Logged on to comment or reply to a post.

  1. Marc Bernard

    Hi Andreas,

    very nice blog, well done! I like visualizing code changes via the heat maps a lot. It does make a lot of sense for custom code. For the generated programs one has to be a bit careful with the approach since there could be several reasons why code is generated again resulting in different code. For example, the implementation of SAP Notes (programm corrections) or support packages can trigger the regeneration. Sometimes the code related to all objects like BW queries is regenerated (for example if there was a bug in the code template), but the code generation is triggered only when the BW query is actually executed by the user (which might be months later).

    In general, I would recommend to focus on the custom code that is embedded in the generated code. It’s not always easy to identify but for example for transformations, one find the custom code in the “compute…” methods. The scan logic would become quite a bit more complex since you would first have to identify the type of BW object related to the code and then apply certain filters.

    Just curious, do you have an example showing the info available when “hovering”? How does the drill down work? Maybe you record a short demo video. Would be great to see.

    Thanks again for sharing,
    Marc Bernard
    Product Management SAP EDW (BW/HANA)

    (0) 
  2. Andreas Göb Post author

    Hi Marc,

    Thanks a lot for sharing you experience. With regards to the bulk regeneration of code I’m not sure whether I completely understand the distinction you make between instant code changes for a large number of objects and code generation triggered for individual objects that is triggered later. Do you say that if a template is changed, the generation of ABAP source code for the respective objects is delayed until actually used, or is just the bytecode not generated before an object is used? Our analysis is based on the source code, so this would make a big difference.

    Thanks a lot for suggesting to consider the relevant methods only (i.e. the ones containing custom logic). I already tried to go in that direction, but I could not find a comprehensive list of these methods, so I decided to first take the simple approach and see how far we can go with it. With regards to treating objects differently based on their type of BW object, we already have some custom parts, so it should not be that hard to add custom filters. Actually, the ConQAT infrastructure we use internally is very flexible in this regard. So if you could point me to some information about which methods contain custom logic, I would appreciate that a lot.

    In order to get a feeling of what information you can see on the dashboard, I blurred a screenshot from an actual instance. When pointing at a rectangle, you would usually see the name of the processing block, the object it is contained in, the package that object lives in, and the number of statements the block contains. In the BW case, we have the name of the generated program, the object type and type-specific data to identify the object:

    /wp-content/uploads/2015/05/query_701793.png

    For Queries this is the InfoCube and the component, while for DTPs and Transformations this is the source object (name, type, and subtype) and the target object. Currently, there is no immediate drill-down directly in the dashboard, but we are working on a much more interactive interface.

    Best regards,

    Andreas

    (0) 
    1. Marc Bernard

      Hi Andreas,

      sorry for the delay but I was on paternal leave. It’s not the bytecode, but the ABAP source code that will be regenerated at the time of first usage. At the time of execution of an object the system compares various timestamps and metadata properties to see if regeneration is required. So, yes,  if a template is changed, the generation of ABAP source code for the respective objects is delayed until actually used.

      However, it depends on the type of BW object. On a system wide basis, it’s quite difficult to see which generated code is up-to-date and which not (unfortunately). For your solution, I think it’s ok to assume that the analyzed code is up-to-date.

      Best,
      Marc
      Product Management SAP EDW (BW/HANA)

      PS: Please send me an email if you like more details.

      (0) 

Leave a Reply