Code Scanner goes ADT

[stockbal]

Introduction

Dear community,

Since I discovered and learned to love the ABAP Development Tools (ADT) it always bothers me when I need a tool during development and it is not available in ADT.

This time I needed to do some code clean-up and search my code for specific string occurrences and replace them with constants. So...,  Where-Used is out 😒. Some kind of string based search was needed.

Some might say:

"But there is something in ADT, the ABAP Source Search!".

For sure, the feature is great, if you have some pretty good arguments for the project manager to approve the additional costs for the development system. (explanation: the HANA DB needs a lot of extra memory for the index creation in the source code tables and therefore a bigger AWS instance was required). Unfortunately I had no such luck.

What to do? Well, guess I have to jump back into SAP GUI and look for an adequate transaction or program to do the work.
I found two programs which could do the job:

• CODE_SCANNER (Transaction)
  
  + supports package resolution
  
  
  
  - no search in $-packages under $TMP
  
  
  - no RegEx support
  
  
  - only 1 String as input allowed (or 2 via logical AND)
  
  


• RS_ABAP_SOURCE_SCAN (Program)
  
  + RegEx or Substring search
  
  
  + Multiple patterns allowed via SELECT-OPTION
  
  
  + More granular selection of report sources
  
  
  + Result grouping in Output
  
  
  
  - no package resolution

In the end, my choice fell on RS_ABAP_SOURCE_SCAN, as it just gave me more options - not the least of which was RegEx. If the program was started from ADT and a match was selected the proper editor was actually opened in ADT, so all in all a pretty good compromise.

But after maybe the 50th time I started the program and I saw the SAP GUI, it started to bug me. And as I needed some new project in my free time anyway, the choice was clear. A new ADT plugin had to be built 😊.

From SAP GUI to ADT

When I thought how to move the functionality of a - potentially long running - ABAP program to ADT I encountered some issues early on.

For one, the possibility to show the current progress to the user, a feature that can be done very easily in Dynpro programming. But in ADT a call to the ABAP backend is done via RFC. Once you are there it is basically over. Only after the RFC call is done the UI can be notified.

Another pain point is the timeout of a dialog process. In SAP GUI one has the possibility to create a background job. When the end result of the program is to display the results in an ALV it is automatically written to the spool so you don't necessarily need to persist the data to view it after the job is done.

So, how to solve these problems? To prevent the timeout from occurring I first thought about restricting the number of objects that can be searched during a single search query. A good option if you always know the scope of your search (i.e. package, application component, etc.). But if you have no idea in which object your string/pattern could be, you would have to set the maximum number of objects to a high value which again would run the risk of getting a timeout. But at the time I had no other solution so I went with this approach.

In regards to the progress indication, again I was out of ideas, so, moving on.

Parallelization to the rescue

Right from the start of development I wanted to include the option to parallelize the search as this was a feature that I was sorely missing from the currently available programs for code search. And I also thought that this could solve the timeout issue.

It gives a nice boost and allows to set the maximum number of objects even higher. But, what happens if you set it to a high number and parallelization is not possible as not enough work processes are available at that time. Then it runs sequentially with a very high chance of a timeout.

So, again not the solution that would solve all my timeout problem. I decided to restrict the maximum number of tasks for parallelization to 10, even if the used server group would allow much more than that.

Persistency is your friend

Whilst implementing the parallelization feature I came up with the idea to check if it would be possible to do it not in the backend but in the UI. I did not really pan out because the search is already running in a background thread but during this time I finally came up with a very simple solution to solve both my timeout problem and the progress indication I was still missing 👌.

The solution is persistency. I kind of gave it away in the heading there, but I did not yet tell you what exactly I am persisting. Because, of course I do not really want to persist the code findings themselves.

I decided on persisting the search scope. So the first request after you are pressing Search is to create a simple representation of the search scope and return the Key of this entry and the determined object count back to the UI. This also has the nice side effect that I can tell the user if the scope found no objects at all.

So after I determined the scope I can tell the background job in ADT how much work I am planning to do. That is needed for the correct progress indication. Then, it is basically just a simple loop in which a new search request with the current offset and package size is sent to the ABAP back end. The package size is fixed to 100 which insures a good response time and progress indication. When a request is done and results were found the Search Result View in eclipse is updated immediately and you can start working with them right away, whilst the query is continuing to send requests to the backend.

Progress indication in the Code Search

With this approach you can easily search your whole ABAP system without the fear of getting a timeout on the ABAP side. Your PC of course has to be awake for eclipse to be able to send new requests.

In an earlier paragraph I wrote that the package size is restricted to 100. Where does that leave parallelization you may ask. That's also solved really nicely with the new persistent scope solution. The activation of parallel processing is done via the project specific settings in eclipse, partly because of the optional server group you can assign to be used (see transaction RZ12).

Parallelization Settings

As you can see there is a setting to specify a number of objects to search if parallel processing is possible. So, if enough tasks are available a single search request will process the package size from the settings and if not the fallback will be 100.

About Performance

As most users would want their results fast, performance was always in the back of my mind. To check the runtime I wanted an easy way to do it from the UI, so I created a simple dialog which would record some useful information during an ongoing search.

The plan was not move it into the final product, but in the end I still did it as it gives some indication why the progress indicator remains very long at a certain percentage.

Query Runtime Information

I think I have bored you enough with implementation details and you are dying to see the final result, right? 😉

The Plugin Features

So here we are. A fully functioning Code Search, supporting all the commonly used source types from Function Groups to even the new Behavior Definitions, and compatible with Release 7.40SP09 (no HANA required).

I will not describe all the features here as they can be read in detail in the integrated help in eclipse anyway, so I will concentrate only on the most interesting ones.

As the name ABAP Code Search suggests, placing the feature in the eclipse Search Dialog (Ctrl + H)  was pretty much a no-brainer.

Search Dialog

Except for a few exceptions the handling of the dialog should be self explanatory. Right on the top is the most important input field, where you can enter your search pattern(s).

If you do not check the option Single Pattern Mode, each line in the pattern input will be viewed as single pattern which you are looking for. There is no restriction to how many patterns you can enter, but keep in mind, the more patterns you enter the longer the search will probably take. I did not really try to bring the search engine to its knees by entering too many patterns, so feel free to test that 😜.

The Creation Date Filter

You can restrict your search on objects that were created in a certain date range. I saw SAP implemented this in the ADT Quick Search Dialog (Ctrl+Shift+A) with separate filters called created (creation year), month (creation month) and date (creation day). Which I thought not very flexible.

Personally I really liked the relative date patterns in the Date Picker control of SAP UI5, so I tried to adjust the patterns so that they would work in the filter input. This approach offers a great freedom in entering date filters.

The following patterns can be used with the created-Filter of the Code Search:

• Relative dates: today, yesterday, last-week, last-month, last-year


• Numeric Relative dates
  e.g. 2-weeks-ago, 15-days-ago


• Day/Month/Year patterns: <dd>.<mm>.<yyyy>
  e.g. 16.4.2021, 1.1.1999


• Day/Month patterns: <dd>.<mm>
  e.g. 1.2, 10.11


• Month/Year patterns:  <mm>.<yyyy>
  e.g. 4.2020, 10.2021


• Range patterns:  <pattern1>...<pattern2>
  e.g. 1.1...yesterday, 15.12...3-weeks-ago

Additionally the modifiers <, <=, >and >= can be used to prefix the date patterns (this excludes range patterns)
e.g. <=today,>11.1.

Also worth mentioning, the relative dates like last-week don't denote a range but a single day in the past.

Regular Expressions

Like the program RS_ABAP_SOURCE_SCAN the ABAP Code Search also allows the usage of regular expressions. As you may know the RegEx engine in the ABAP kernel can be brought to its limits very quickly if the expression becomes too complex. So you may have also encountered the exception CX_SY_REGEX_TOO_COMPLEX in the past.

In the ABAP Release 7.55 SAP has introduced a more rebust syntax called PCRE (Perl Compatible Regular Expressions). There is a very interesting blog series about this if you are interested.

Of course, not everyone is lucky enough to develop software on such a new platform, but as SAP is saying that PCRE is generally more robust and better performing than the old POSIX standard, the ABAP Code Search is using PCRE by default if it is available.

Sequential Matching

This special search mode came to be, partly because of the restrictions of the RegEx engine in older ABAP releases and partly because writing regular expressions is not always that easy. Even I am far away from being an expert.

To make it easy to understand what this search mode is all about I will explain it with an example:

Let's look at following code snippet, a SELECT statement inside a nested LOOP.

LOOP AT entries ASSIGNING FIELD-SYMBOL(<entry>).



  LOOP AT <entry>-sub_entries ASSIGNING FIELD-SYMBOL(<sub_entry>).

    SELECT SINGLE author

      FROM tadir

      WHERE obj_name = @<sub_entry>-obj_name

       AND object   = @<sub_entry>-object

      INTO @<sub_entry>-object_author.

  ENDLOOP.



ENDLOOP.

A constellation which can become problematic in regards to performance. I guess this specific example can also be found by running an ATC check on the object. I only took it to demonstrate some of the options the Sequential Matching is offering.

So, to find such a SELECT with the Code Search could certainly be done with the PCRE and multiline search (I am actually guessing as I have not tried), but without RegEx it is downright impossible.

With Sequential Matching however it is very easy. By writing the following sequence in the pattern input you will get matches for the above code sample - only if they exist in your searched code of course.

·loop·
(#exclude)·endloop
·loop·
(#exclude)·endloop
(#m-start)·select·
(#m-end).

What's happening here? Well, generally speaking, the sequential search returns only matches if all the given patterns are matched in the specified order. For that the source code is searched row by row and subsequent patterns will be looked for at the offset of the match the previous pattern.

If no control sequences were entered (like (#exclude) in the sample) the match will span from the match of the first pattern to the match of the last pattern.

If you want to know more about this and the rest of the features you are going to have to install the plugin 😎

Installing the plugin

Prerequisites

To install and use the plugin you first have to do the following 2 steps:

1. Install ABAP Development Tools Plugin https://tools.hana.ondemand.com/#abap


2. Make sure the following abapGit Repository https://github.com/stockbal/abap-code-search-tools is installed.

References

You can install the plugin via the Update site https://eclipse.devepos.com/latest/ or via the eclipse marketplace https://marketplace.eclipse.org/content/abap-code-search.

Final Words

I hope you find this new addition to the family of Open Source plugins for ADT useful. For me personally it was a fun project and I am kinda sad to see it end, but I still have some ideas for new plugins, so everything is well 🙂.

Cheers

Ludwig