While there are quite some good documents about the setup of the ABAP Test Cockpit (ATC) on SDN (cf. http://scn.sap.com/docs/DOC-32791 and http://scn.sap.com/docs/DOC-32628) I haven't seen any experience reports about a roll out of ATC yet. Therefore I decided to blog about my current experiences in rolling out the ATC in our development organization.
Step 0: Some Background
Before starting to describe what we did in our ATC roll out I want to give you some background about the environment of the roll out. At my company we are managing and maintaining several SAP system landscapes for different customer. A typical customer landscape consists of a SAP CRM, a SAP IS-U (ERP) and a SAP BW together with several non-SAP systems (e.g. an output management system and an archive system). In addition to that we have a central development which is used to develop core functionality and distribute these across the customer systems. These core functionalities are typically developed in our own namespace. Therefore, each of our customer system contains a set of custom development in the customer namespace and a set of developments in our own namespace.
The second important aspect of our environment is the diversity of developers developing in the system. Firstly, we have a core development team. This team consists of people with a deep knowledge around software development and mostly some formal training (e.g. a computer science degree) in the area. Secondly, we have a team of functional consultants with a wide range of development skills, ranging from some basic ABAP knowledge to very deep knowledge. And finally we usually have several external consultants developing in the different customer systems as well.
As you might have guessed the result of this environment is a quite diverse code base containing anything from well designed, reusable components to unmaintainable one-time reports.
Step 1: Analysis of our Custom Code
The first step I took in order to roll out ATC was to perform a first check run using a default check variant in the customer system with the largest code base as well as in our central development system. The result of this first analysis was quite disillusioning. The first run of the default check variant of the ATC across this code base resulted in roughly 700 priority 1 errors, 2500 priority 2 errors and nearly 10.000 priority 3 errors.
Step 2: Discussion within the Core Developer Team
The next step was to discuss the check results with the core development team. This discussion basically consisted of two parts.
Firstly, when I presented the tool everyone agreed that it would be very useful and we should use it. When we then had a detailed look at the check results from the two systems they were not that positive any more. The main criticism was around the errors raised by the ATC. Especially some of the more common errors lead to quite some discussions whether the reported error was really an error or rather a false positive. Furthermore, it turned out that some of the default checks simply are not valid in our system landscape. An example of such a check is the Extended Program Check that checks for conditions that are always false. In the context of SAP IS-U the pattern "IF 1 = 2. MESSAGE..." is used extensively throughout the SAP standard. Consequently, it is also widely used in our custom code. However, the Extended Program Check reported each of these if statements. There reason is, that the check only allows for the pattern "IF 0 = 1. MESSAGE....".
Secondly, we discussed extensively how we should approach the large number of issues in our code base. It was obvious that we wouldn't be able to fix all reported issues. This would also have been not very sensible. One reason is that a lot of the programs for which issues were reported might not be in use any more.
As a result of the discussion we decided to:
- define a custom check variant including only the relevant checks
- define a custom object set.
Step 3: Definition, Testing and Rollout of a custom Check Variant
The next step we took was the definition of a custom check variant. The process of the definition of the custom check consisted of several parts. We started by defining an initial set of checks that we wanted to use. Furthermore, we adjusted the priorities of the checks to our need. It's pretty obvious that each error that might cause a short dump needs to be error of priority one. However, with other checks the correct priority is not that clear. Consider for example the check for an empty where clause in a select. A program containing such a statement might cause severe performance problems in production if it is executed on a large table, nevertheless it might be fine in a small correction program that is only executed once. Last but not least we modified some of the default checks (cf. the IF 1 = 2 pattern mentioned above) to suite ore needs. Unfortunately, the modification of the default checks required a modification of the SAP standard in some cases.
After the initial definition of the check variant we set up daily check runs in the QA system including the replication of the results into the development system. With this set up we worked for some weeks and iteratively refined our default check variant.
Step 4: Definition of a custom Object Set
Besides the executed checks we also needed an approach to cope with the large number of errors present in our code base. For this we decided that from now on we only wanted to transport objects into the production system without any priority 1 or priority 2 errors. However, we also decided that we didn't want to correct legacy code unless we were modifying it anyway (for example as a result of a bug fix or new feature request). Therefore we created a custom object set and a custom object collector. The custom object collector will only ad objects to the object set if it has been modified after a certain date. This way we were able to get check results only for new or recently modified objects.
Note that this approach has an important drawback. If for example the interface of a method is changed (e.g. by adding a additional required parameter) this might cause a syntax error in some other program using the class. However, with our custom object collector ATC will not be able to find this error as the program using the class itself is not changed. Nevertheless this was the approach we choose to cope with the large amount of legacy code.
Step 5: Rollout across all Developers
After the core development team had been working with the described set up for a while we were quite comfortable with the results that the ATC produced. Therefore we decided to roll out the ATC to all developers working in our system. This was done by informing everybody about the ATC as well as setting up the execution of the ATC checks upon release of transport request. Note that we for now only executed the checks upon release of a transport but did not block any transports because of ATC errors.
As a result of executing ATC upon the release of a transport request basically every developer was immediately using ATC, even if they had not integrated it into their workflow yet. This proved very successful, especially with the less experienced developers. As the ATC provides useful explanations together with each error it resulted in quite some discussion and learning regarding good ABAP code that wouldn't have happened otherwise.
Summary and Next Steps
After working with the described set up now for a few weeks the roll out of ATC proved quite successful in or development organisation. Especially the detailed documentation of the ATC errors help to improve the knowledge across the organisation. With respect to the roll out I think involvement of the core developers from the very beginning was very important. Only by agreeing on a set of ATC checks, sometimes only after a few discussions, everyone accepts the raised errors and fixes them. If we would have simply used the default check variants without the adaptations mentioned above I don't thinks the ATC would have been accepted as a tool to improve the code quality (e.g. due to a large number of false positives).
The next step we will take is the roll out of the ATC exemption process in our development organisation.The reason is that we already noticed that some priority 2 errors can't be fixed due to different restrictions (e.g. usage of standard SAP functionality in custom code that leads to error messages). Therefore we need the exemption process in order to remove the errors in those special cases. Furthermore, I see the exemption process also a prerequisite to disable the release of transport request as long as ATC errors are present.
Finally, I'd be happy to discuss experiences with other ATC users.
Christian
