While there are quite some good documents about the setup of the ABAP Test Cockpit (ATC) on SDN (cf. http://scn.sap.com/docs/DOC-32791 and http://scn.sap.com/docs/DOC-32628) I haven’t seen any experience reports about a roll out of ATC yet. Therefore I decided to blog about my current experiences in rolling out the ATC in our development organization.

Step 0: Some Background

Before starting to describe what we did in our ATC roll out I want to give you some background about the environment of the roll out. At my company we are managing and maintaining several SAP system landscapes for different customer. A typical customer landscape consists of a SAP CRM, a SAP IS-U (ERP) and a SAP BW together with several non-SAP systems (e.g. an output management system and an archive system). In addition to that we have a central development which is used to develop core functionality and distribute these across the customer systems. These core functionalities are typically developed in our own namespace. Therefore, each of our customer system contains a set of custom development in the customer namespace and a set of developments in our own namespace.

The second important aspect of our environment is the diversity of developers developing in the system. Firstly, we have a core development team. This team consists of people with a deep knowledge around software development and mostly some formal training (e.g. a computer science degree) in the area. Secondly, we have a team of functional consultants with a wide range of development skills, ranging from some basic ABAP knowledge to very deep knowledge. And finally we usually have several external consultants developing in the different customer systems as well.

As you might have guessed the result of this environment is a quite diverse code base containing anything from well designed, reusable components to unmaintainable one-time reports.

Step 1: Analysis of our Custom Code

The first step I took in order to roll out ATC was to perform a first check run using a default check variant in the customer system with the largest code base as well as in our central development system. The result of this first analysis was quite disillusioning. The first run of the default check variant of the ATC across this code base resulted in roughly 700 priority 1 errors, 2500 priority 2 errors and nearly 10.000 priority 3 errors.

Step 2: Discussion within the Core Developer Team

The next step was to discuss the check results with the core development team. This discussion basically consisted of two parts.

Firstly, when I presented the tool everyone agreed that it would be very useful and we should use it. When we then had a detailed look at the check results from the two systems they were not that positive any more. The main criticism was around the errors raised by the ATC. Especially some of the more common errors lead to quite some discussions whether the reported error was really an error or rather a false positive. Furthermore, it turned out that some of the default checks simply are not valid in our system landscape. An example of such a check is the Extended Program Check that checks for conditions that are always false. In the context of SAP IS-U the pattern “IF 1 = 2. MESSAGE…” is used extensively throughout the SAP standard. Consequently, it is also widely used in our custom code. However, the Extended Program Check reported each of these if statements. There reason is, that the check only allows for the pattern “IF 0 = 1. MESSAGE….”.

Secondly, we discussed extensively how we should approach the large number of issues in our code base. It was obvious that we wouldn’t be able to fix all reported issues. This would also have been not very sensible. One reason is that a lot of the programs for which issues were reported might not be in use any more.

As a result of the discussion we decided to:

  • define a custom check variant including only the relevant checks
  • define a custom object set.

Step 3: Definition, Testing and Rollout of a custom Check Variant

The next step we took was the definition of a custom check variant. The process of the definition of the custom check consisted of several parts. We started by defining an initial set of checks that we wanted to use. Furthermore, we adjusted the priorities of the checks to our need. It’s pretty obvious that each error that might cause a short dump needs to be error of priority one. However, with other checks the correct priority is not that clear. Consider for example the check for an empty where clause in a select. A program containing such a statement might cause severe performance problems in production if it is executed on a large table, nevertheless it might be fine in a small correction program that is only executed once. Last but not least we modified some of the default checks (cf. the IF 1 = 2 pattern mentioned above) to suite ore needs. Unfortunately, the modification of the default checks required a modification of the SAP standard in some cases.

After the initial definition of the check variant we set up daily check runs in the QA system including the replication of the results into the development system. With this set up we worked for some weeks and iteratively refined our default check variant.

Step 4: Definition of a custom Object Set

Besides the executed checks we also needed an approach to cope with the large number of errors present in our code base. For this we decided that from now on we only wanted to transport objects into the production system without any priority 1 or priority 2 errors. However, we also decided that we didn’t want to correct legacy code unless we were modifying it anyway (for example as a result of a bug fix or new feature request). Therefore we created a custom object set and a custom object collector. The custom object collector will only ad objects to the object set if it has been modified after a certain date. This way we were able to get check results only for new or recently modified objects.

Note that this approach has an important drawback. If for example the interface of a method is changed (e.g. by adding a additional required parameter) this might cause a syntax error in some other program using the class. However, with our custom object collector ATC will not be able to find this error as the program using the class itself is not changed. Nevertheless this was the approach we choose to cope with the large amount of legacy code.

Step 5: Rollout across all Developers

After the core development team had been working with the described set up for a while we were quite comfortable with the results that the ATC produced. Therefore we decided to roll out the ATC to all developers working in our system. This was done by informing everybody about the ATC as well as setting up the execution of the ATC checks upon release of transport request. Note that we for now only executed the checks upon release of a transport but did not block any transports because of ATC errors.

As a result of executing ATC upon the release of a transport request basically every developer was immediately using ATC, even if they had not integrated it into their workflow yet. This proved very successful, especially with the less experienced developers. As the ATC provides useful explanations together with each error it resulted in quite some discussion and learning regarding good ABAP code that wouldn’t have happened otherwise.

Summary and Next Steps

After working with the described set up now for a few weeks the roll out of ATC proved quite successful in or development organisation. Especially the detailed documentation of the ATC errors help to improve the knowledge across the organisation. With respect to the roll out I think involvement of the core developers from the very beginning was very important. Only by agreeing on a set of ATC checks, sometimes only after a few discussions, everyone accepts the raised errors and fixes them. If we would have simply used the default check variants without the adaptations mentioned above I don’t thinks the ATC would have been accepted as a tool to improve the code quality (e.g. due to a large number of false positives).

The next step we will take is the roll out of the ATC exemption process in our development organisation.The reason is that we already noticed that some priority 2 errors can’t be fixed due to different restrictions (e.g. usage of standard SAP functionality in custom code that leads to error messages). Therefore we need the exemption process in order to remove the errors in those special cases. Furthermore, I see the exemption process also a prerequisite to disable the release of transport request as long as ATC errors are present.

Finally, I’d be happy to discuss experiences with other ATC users.

Christian

To report this post you need to login first.

12 Comments

You must be Logged on to comment or reply to a post.

  1. Thomas Fiedler

    Hi Christian,

    thanks for sharing your experiences with the ATC. Do you also have the ABAP in Eclipse integration already in scope? You can run ATC checks manually in AIE and you can configure the Feed Reader in order to see the check results of the ATC runs of your Q-systems. Any experiences with that?

    Regards,

    Thomas.

    (0) 
    1. Christian Dr. Drumm Post author

      Hi Thomas,

      I’ve been playing a little bit with the AIE integration of ATC. Most of my colleagues are not using AIE yet due to constraints in our landscape (wrong NW version, missing features due to that, etc). So I’ve got not much experience in this area.

      Personally, I like the feed reader very much. Using it to detect dumps in production systems as well as to receive ATC check results.

      Christian

      (0) 
  2. Florian Wunderle

    Hi Christian!

    good post! We had a similar approach with our rollout. Since we have a quite intensive rollout and several project ongoing we even decided to switch on ATC for transport release and use exemptions for all existing code from the time before ATC (appx 3 years of development).

    There are still some difficulties when it comes to exemptions. It is e.g. not possible to assign them to a group). What we are also waiting for is the integration into ChaRM with Solman 7.1 SP10 to collect results of both implementation and maintenance line.

    /Florian

    (0) 
  3. Sergio Ferrari

    Hi Christian,

    thanks for sharing your experience.

    Could you tell something more about the sentence “Unfortunately, the modification of the default checks required a modification of the SAP standard in some cases.“.

    CI Message Priorities can be partially adjusted via the /nSCI transaction.

    I think Extended Program Checks related priorities could be adjusted via table SLIN_DESC and OSS note 1952207 – Mapping SLIN message priorities in code inspections introduced a new logic in the Mapping between the layers.

    What about your approach?

    BR

    Sergio

    (0) 
    1. Christian Dr. Drumm Post author

      Hi Sergio,

      you are right, it’s possible to adjust the message priorities without the need to modify the SAP standard.

      The problem was rather that some of the checks didn’t quiet suite our need. One example is the check for unreachable IF clauses. Here, SAP doesn’t raise an message for the pattern “IF 0 = 1. MESSAGE…”. This approach is frequently used to enable cross references from message classes. However, across SAP IS-U the pattern “IF 1 = 2. MESSAGE…” is used, even in the SAP standard. So we have got a lot of custom code that contains this pattern. As a consequence we received many error messsage in the initial check runs. Although I don’t agree with the approach (instead of IF 1 = 2. MESSAGE…” I’d always use “MESSAGE … INTO …” to archive the same result) we decided to modify the source code of the check. After this modification “IF 1 = 2” is now treated like “IF 0 = 1”, i.e. no error message is raised.

      There was also an other check were we did a similar modification. However, I currently can’t remember any more which check it was…. 😛

      Christian

      (0) 
  4. Mahadevan Venkata

    Hi Christian,

    Nice post and thanks for sharing your experience.

    Regarding modification to the SAP standard variant, I suggest you copy the default variant, create/assign your own checks and then use them in ATC.

    Best Regards

    Mahadevan

    (0) 
    1. Christian Dr. Drumm Post author

      Hi Mahadevan,

      thanks for the reply.

      Copying the SAP standard check variant is exactly what I did. It allowed me to only call certain checks in ATC.

      Your are right, that I could have created copies of the existing checks (e.g. the one for IF 0 = 1.) and adapt them to our needs. However, as the necessary modifications where rather small I choose to modify the SAP code. If you need completely new checks or major modifications to the standard checks would be required, creating custom checks (from scratch or by copying existing ones) is definitely a viable option.

      Christian

      (0) 
  5. Alexander Ryabov

    Hi Christian,

    could you please clarify something?

    1. You said youa are running ATC daily checks on QA system.

    Does this mean that you transport changes daily to QA?

    2. The second thing I couldn’t catch is: it seems like you transport all developments to QA from DEV no matter if there were some errors during TR release.

    Well, than it means that first the code goes anyway to QA and later you start bug fixing phase.

    Why don’t force TR release untill errors are fixed and keep everything in DEV system?

    Thank you.

    (0) 
    1. Christian Dr. Drumm Post author

      Hi Alexander,

      sure, at least I will try to.

      Your assumption, that we transport developments into QA daily is correct. In fact we are even transporting developments into QA several times a day. Developers also do their developer tests in the QA system. Therefore, whenever a development is finished, the developer transports its changes (as a transport of copies) into the QA system. Therefore, it makes sense for us to run the checks daily in QA. Furthermore, ABAP has no such function as a build and a very weak package concept. If you change a function module or class you can never know where it has been used and if you might break things with your change. The daily execution of the checks in QA also gives us visibility of such issues. If a change in one area will break something elsewhere, we will be automatically notified the next day.

      Regarding transporting objects with issues to QA (and even to the production system); In a perfect world I would agree with you, that everything needs to be fixed in DEV before anything is released into QA. However, as mentioned we have a large code base with productive code that has grown over the years to manage. While the check cockpit might raise certain issues the code nevertheless is tested and in productive use. So whenever a ne feature is requested or a bug is fixed we are in the dilemma how much to change. We agreed in our team, that we will always fix all prio 1 errors. Regarding prio 2 and 3 issues it really depends on the development at hand. For completely new developments we aim for zero issues before releasing to QA. However, if a minor bug fix needs to be done in a large function, we might still release it to QA, and subsequently to production, even if it still got prio 2 and 3 issues in unchanged areas of the code. From my point of view this approach was successful for two reasons. First, it enabled developer acceptance. Forcing zero issues from one day to the other would have resulted in a “I didn’t break it so why should I fix it?” mind set and therefore bad acceptance. Second, and most importaned it allowed to perform the roll out of the ATC checks without the need of an extensive cleanup phase first. This is especially important in small development organizations where the resources for a complete clean up would simply be not available.

      Hope this helps to clarify a little bit.

      Christian

      (0) 
      1. Alexander Ryabov

        Thank you Christian for such detailed explanation.

        Now it seems to be an issue how to observe the quality on all systems…

        I would suggest you to have a look on to CCLM tool, which is based to Solution Manager.

        (the higher the Service Pack – the better).

        From SolMan you can get the transparancy of where your code was found, what was the last ATC check for this object and wether is used or not.

        I also suppose you are using CHARM in SolMan for dealing with such amount of trasnports daily.

        Lately, you can assign responsible person for the object in CCLM or even an organization.

        And much more like dashboards, etc…

        Here: Custom Code Management – SAP Solution Manager – SAP Library

        and Here: SAP Solution Manager WIKI – Custom Code Management – Solution Manager – SCN Wiki

        (0) 
  6. James Swanson

    I know this is years late but it may help someone.  You said . . .

    “Last but not least we modified some of the default checks (cf. the IF 1 = 2 pattern mentioned above) to suite ore needs. Unfortunately, the modification of the default checks required a modification of the SAP standard in some cases.”

    These are the check variants from SCI right? So you can copy them from “DEFAULT” into your own copy and then change which variant SAP uses as the default in table SCICHKV_ALTER. and TRANSPORT is the one for transport inspection.

    (1) 

Leave a Reply