Skip to Content

At the recent SAP TechEd conference in Las Vegas, Michal Krawczyk from Int4, a fellow mentor, conducted a session (NET38702 – which will be repeated in Teched Barcelona), running test scripts in SAP PI/PRO, and I was amazed by how many people showed up! At first I couldn’t understand why such a huge number of people would attend — it’s not like testing is a very exciting topic. But when the discussion got started, I started to get it.

/wp-content/uploads/2016/09/20160920_160230_1045882.jpg

Despite the fact that testing is a problem that confronts everyone in the field, very few people are working on testing tools. In other fields, there is a culture of test-driven developments, but this isn’t something we have done much in PI/PRO. So TechEd attendees came to learn about what is being developed in the area of SAP PI/PROtesting.


The basic problem is that although we want to be able to do testing, current methods are inadequate to cover the use-cases that exist. When Michal Krawczyk asked for a show of hands of how many people were working on testing tools, only a handful of people raised their hands, so I thought I’d share their research in this post.


One group of developers was conducting manual tests. They had saved a couple of test cases as test instances in Message Mapping or maybe Operation Mapping, and once they had made changes, they would call up these messages to see if they would still run. The problem with this approach is that since you are using older messages that haven’t changed much, it is difficult to catch the unknown changes you don’t know to look for. This makes for a somewhat limited test, but it will still give valid feedback if the work is correct.


Some other people were using XMLSpy from Altova to do some tests on their server. I think you need Enterprise version to do tests this way.


Another guy I spoke with was using Ready API from Smart Bear, the guys that are making SOAPUI that we all are using. With this tool, they are able to set up test cases; for example, it allows a bit of XML to be sent through the system, then downloads the message, and generates an assertion that allows us to specify the values we want to see.


One group is using a Computer Associates or HP testing tool to conduct end-to-end testing. This is how it works: First, log into the third-party system, pressing all the keys to generate a script that goes through the screen; next, fill in the relevant data, and finally, complete the transaction, and the message should be sent through the PI/PRO to the ERP system. At that point, make an assertion that the order was created, and the system should return a total amount, etc. This method would cover the full process end to end.


One of the challenges with this kind of tool is that it requires a lot of time to set up the test cases. This means that whenever you want to do a test, you first have to go through all these test cases, make sure they match, and update them. Only once they’re running well, are you ready to start the actual test. Another potential problem could be that since you have to go through so many steps, you may limit the complexity of the example transaction (for example, you might not create that complex purchase order), or the number of tests you run (you might run only one or two cases, when you really should run ten).


There was also a coding session by SAP showing how they were doing their regression testing. In this case, they had a virtual environment, which means they were able to create scripts that were built in the PI system from scratch. These scripts were deployed to an Amazon Cloud instance, which could then be used to run different scripts, like adding or calling the webservice and getting a response back. Developers could then gather data on how well that task had been performed, how long it took to do ten calls, then a hundred calls, and so on. They were also able to vary the size of the message to see how well the message scaled on all different levels. This is obviously helpful for testing system performance, because it allows testers to ask questions like ‘Did we add anything that just makes the database expand?’ or ‘Did the performance to suffer?’  For most customers, this method would not be particularly useful because they would first need to set up the framework and collect the business cases.


Michal Krawczyk and his company Int4 have developed an interface regression testing tool – IFTT which is able to regression test both SAP PI/PRO as well as SAP backend system interface (user exits, ABAP enhancements). Due to the fact that the tool is tailored for SAP landscape you can easily create test cases from any existing IDOC or ABAP Proxy message by selecting the message ID (IDOC number or GUID) only. There’s also no need to create any assertions as IFTT is comparing the data on the SAP database level (using SAP tables). By utilizing this tool both the developers and anyone responsible for interface testing should be able to catch any unwanted change before it makes it to production landscape, allowing them to pinpoint a potential production problem. Michal Krawczyk said they at one customer they have tested 30 different scenarios with total of around 10.000 different test cases (messages). Of course, that’s a bit of a different scale than if you’re just sending in five or six different test cases per scenario — with this you’re going to cover a lot more ground and be sure that the mapping and the whole interface on the SAP backend system is working.


My company, Figaf, has also made a SAP PI testing tool that makes it easy to record and test messages from the sender module to the receiver model chain. I believe the challenge with these kinds of systems is to find a way to catch the message and test it all the way through the process, while keeping the method as simple as possible. Figaf meets this challenge by adding two modules to our production system, which allows us to replicate the traffic on that development system, so we can check that all modules and routing capabilities stayed the same.


With this tool, we’re able to get a lot better test coverage. In ten clicks, I can set up a test case for a hundred new documents and be able to process them in the easiest way. In contrast, users of some of these other tools might need to use a third party system or work through various different steps before being able to run the tests. At this point, the assertion the system receives is the original output document, so I can expect the same on my test system. Then, as you proceed, you can ignore some elements (for example, if we know this timestamp is different, or this system name is different), so you only see the important differences. The tool supports comparisons in EDIFACT, X12, Text and XML. It does not test adapters or backend systems, but it does make it a lot easier to test all mappings, modules and routing in your system.


I have heard of a system from X-Case that is doing something similar, though I haven’t tested it to see what it can do or how it works. Also, IBM purchased Green Hat now Rationale Integration Tester, which is one of these more complicated environments to set up and run, I’m not sure on how well it works with PI/PRO.


The the last tool I want to cover is one created by my old colleague Morten Wittrock about 10 years ago, called the Detroubulator. This tool was able to run message mappings, but it was really cumbersome to set up all the test cases.


So that wraps up the list of SAP PI/PRO testing tools that I wanted to cover in this post. If you have other tools you are using, please share them so everybody else can get a better idea of how you are testing and what you are testing. If you are looking to perform tests on your own system,  do check out some of these tools — and let us know which ones helped you the most.

To report this post you need to login first.

8 Comments

You must be Logged on to comment or reply to a post.

  1. Morten Wittrock

    Writing unit tests for Detroubulator was indeed cumbersome, but that’s qualitative testing in a nutshell. You can obviously generate quantitative regression test cases from existing messages (and in fact we did), but they do not tell you whether your stuff works or not, just that it works the same way it did prior to the change prompting the regression test 🙂 There’s value in both approaches, of course.

    (0) 
    1. Bhavesh Kantilal

      If there was ever a need for a use case on the success of the DeTrobulator tool that you had created – do let me know 🙂

      We used if for a EDI Migration project for over 1600 Suppliers with atleast 300 EDI Mappings! We would take the input and output files from our legacy middleware atleast 50 per suppliert, generate the EDI XML’s for these and then use Detrobulator to trigger our PI mappings and then compare the results! The time it saved was outstanding!

      In hindsight it was way ahead of its time – launched in 2006/2007 -days when there were no Integration Directory API’s.. Not sure if the Detrobulator project is still alive but to those those contributed on it – Morten and your team – thank you! 🙂

      (0) 
  2. LUCA INGENITO

    I joined the session NET39346 for “SAP Interface Scenario RGR testing” in Barcelona TECHED and in few minutes I heard the most common issues in our daily life when we talk about regression testing for our IFs.
    It was funny and very well summarized.
    In my company the usual needs we have in terms of RGR test can be categorized as follows:
    1. PO patching projects
    2. PO upgrade projects
    3. ERP upgrade projects (ECC, CRM, HCM, etc)
    4. Monthly enhancements/corrections releases
    5. Roll-ins twice a year
    6. ENHANCEMENT Projects belonging to the Company PROGRAM twice a year
    7. Infrastructure projects
    8. Migration projects (from other middleware to PO… in the future possibly migration to the cloud)

    Any of the above mentioned has different needs and we struggled to automate test for any of the above cases and even our test team purchased a testing tool that can hardly cover the 20% of the above and actually it doesn’t run a simulation close to the real case that would happen in PROD (I mean the tool doesn’t trigger any message in the test system, but apparently works exporting XSDs and simulating the PO behavior).

    Pretty often (if not always) we finished with manual test and obviously engaging the business users and increasing their unhappiness for the missing automation we could offer.

    Proceeding with order, the “different needs” case specific are as follows:
    1. PO patching projects:
    need of testing the inbound/outbound adapters. No data validation required, simply a validation for the B2B of message received from the 3rd party systems in the case of the outbound. –> We obviously have to engage the 3rd party to verify they received successfully the message, but already having a set of test cases for at least 2 IFs per any of our adapter, could have helped a lot instead of me going behind any user or functional team to trigger messages from the backend.
    Having the possibility to simulate a call from 3rd parties for any adapter (not only SOAP calls) would have avoided to contact several suppliers/customers.
    Having finally the possibility to test also Synchronous scenarios would have been perfect.
    2. PO upgrade projects –> here the complexity is higher. The need is to test the most business critical IFs not only at adapter level but also at mapping level.
    Same situation: having the possibility to for example reuse the test cases created during the last roll-in plus the previous ones, would give us the opportunity not to bother too much the users/functional teams and simply retrigger existing idocs, or simulate proxy calls for the outbound scenarios, or retrigger messages received from 3rd parties, etc etc.
    3. ERP upgrade projects (ECC, CRM, HCM, etc) –> same as PO Upgrade, but with a wider scope that I am not sure could be covered by simply a PO RGR tool. Over here the need would be to potentially test also much more backend functionalities (e,g, FMs, BAPIs, etc)
    4. RGR test for monthly enhancements/corrections releases –> this potentially should be an easy one. To create a test case or multiple test cases linked each other that will trigger all IFs that have interdependencies (such as shared MM, shared DT, etc etc) with the one that got a correction or an enhancement –> e.g. 1 IF has 100 value mapping entries in a table, you need to add 1 more entry…In the meanwhile the MM has been touched and reversed plenty of times that even from the history the developers could get confused and reactivate and enhance a wrong mapping version. who will be testing manually the other 100 cases? –> having it automated could give us I would say over 90% of guarantee that we tested what was there before.
    5. RGR test for Roll-ins. –> Similar to the previous one, but obviously with a wider scope. An IF already working for 4-5 different units around the world need to get extended for the latest roll-in. We need to make sure to not break the existing functionalities in the units already running on our SAP global template –> what if we would have the possibility to retrigger the test cases that were running for those specific unit during their own roll-ins?
    6. RGR test for ENHANCEMENT Projects belonging to the Company PROGRAM. –> same as previous point for the roll-ins.
    7. Infrastructure projects –> I am pretty sure that we are not the only one who faced situations where the NTW team, w/o having visibility about the applications impacted by changing port, firewall rules and whitelists, urls, etc etc, simply made those changes w/o proper notifications and suddenly…panic! IFs are NOT working and PO is the root cause… alerting NTW team to get in touch with my team would not be neither enough….plenty of times is hard to identify all IFs impacted by a NTW change….but what if, like in the case 1 for the adapters, we would have the possibility to run 1 test case per each adapter or even better, 1 test case for any of our IF?
    8. Migration projects (from other middleware to PO… in the future possibly migration to the cloud): this is a though one…we have plenty of relationships between different ERPs using other middlewares… then we should start a migration project (e.g. EDI from Sterling to PO) making sure that the enhancement of the template in which we have to include these existing scenarios running on another middleware, would not harm the units and scenarios already running on PO.

    I could think also other cases were an automated RGR would  be beneficial…e.g. a test environment client refresh…how to quickly make sure that the refresh didn’t harm our IFs, and that for example tables that we are calling in backend are still there?
    At the moment the functional teams make the so called “health checks”, but they are usually manuals and several things are forgotten and then….taaak, the IF is not working anymore.

    I am not a fun of tools that could “simulate” what could happen in PO by running something “virtual”… if I could choose something, I would rather prefer to have a tool that can physically trigger messages into my backend and PO test systems where I can really see in the monitoring that something was running, and where the routing conditions in my ICO or Lookup calls could really happen…and that is what I would say could cover a quite wide scope even for B2B scenarios where at least the backend and PI part could be tested and checked with lower involvement of business users.

    Well, that is my experience and related needs….hope that helps to inspire somebody or let him feel less lonely when he will face our same problems 🙂

    Cheers,
    Luca

    (1) 
  3. Ralph Birk

    Hi there,

    we are switching from PI 7.31 to PO 7.5. Now I have to improve my test framework. Up to now I used the Function SMPP_CALL_JAVA_RUNTIME3 in my JUnits. Does anyone know an API for calling an xslt or java mapping program externally on a java only system?

    Ralph Birk

     

    (0) 

Leave a Reply