Personal Insights
Automated E2E testing for SAP is a mouse trap! Don’t fall for it – Decouple instead.
Why Automated E2E testing for SAP projects is a trap?
Automated E2E testing sounds like dream come true for SAP S/4HANA complex transformations (Selective Data Transitions/BlueField or Complex Brownfield projects) but in reality it’s a hidden mouse trap.
On one hand, thoroughly testing your system in a production-like test environments sounds like a great idea but in reality it’s only doable for very simple systems. When we talk about SAP transformation projects like Selective Data Transitions/BlueField or Complex Brownfield projects where we have hundreds of 3rd party systems and even more EDI/B2B partners it start to resemble a Greek myth of Sisyphus (can never be achieved). It’s brilliantly explained in this video from David Farley author of “Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation”.
What makes E2E testing fail for SAP projects?
Imagine having to run a complex SAP transformation program with 30-100 SAP consultants and many more internal resources which need to work together. Now someone comes to you and says the best way to run this program is to enable E2E testing so whatever change the SAP team will do will be automatically validated. While it’s a great idea in reality you don’t add 5-10% of more work to the scope of the project as with 50-100 3rd party systems and 200-500 EDI partners you may actually be doubling or tripling the project scope – were you aware of that? Doesn’t that remind “spinning too many plates”? Are you such a good coordinator and is your SAP project budget owner are of the additional costs?
Image source: https://halfretire.com/why-selling-a-business-is-a-million-dollar-aspirin/
It doesn’t end here I’m afraid, E2E testing setup would be deployed by people who are not involved in the software development process and don’t know the functionality of those 50-100 3rd party systems. Testing teams in such situations tend to be overly cautious and test everything due to concerns about potential issues. This approach can be slow, costly, and not very effective at ensuring high quality. Also the problem is that testing is happening too late in the process to catch and prevent quality issues from occurring in the first place.
Solution – Simulate to Isolate and Decouple your SAP project
As Dave Farley mentions a good test is deterministic and atomic so the key to solve this puzzle is to use simulation to isolate and decouple SAP project from 3rd party systems and EDI/B2B partners as shown in Figure below.
Figure – Simulate 3rd party systems and EDI/B2B partners for SAP projects.
It simplifies the following:
a) scope of the project – you’re only testing the releasable unit of software – SAP system and not anything else
b) we can test every aspect of the releasable unit of software in a much better way instead of testing only vanilla flows with E2E testing
c) by simulating 3rd party systems and EDI/B2B partners we speed up the SAP project as we eliminate the need to coordinate the testing with multiple parties
d) our SAP project can be more sustainable – no need to create matching tiers of test environments of 3rd party systems and the existing ones don’t even need to be online during the testing
This method is a much more organised and structured approach to conducting comprehensive and detailed testing of SAP transformation programs, and it is more efficient and quicker than the E2E alternative.
Where I can learn more about this topic of Simulations/Service Virtualization for SAP transformation programs?
There’s a 3h course on openSAP just on this topic called “Avoid SAP S/4HANA Project Delays with Third-Party Systems Service Virtualization” – In this course we help you understand why SAP-tailored service virtualization is a hidden gem of SAP S/4HANA projects, who can use it and when to use it, and most importantly, how to benefit from it. In addition, you’ll learn how service virtualization can make your projects more sustainable by significantly minimizing their carbon footprint.
Figure – openSAP course on service virtualization
Futher reading
How to run many SAP S/4HANA rollouts at the same time? Use project tracks!
While it might not be applicable for "every" single process in an implementation: Thanks for pointing out a very promising approach to overcome E2E tests in S/4HANA implementation projects. Definitely worth the time to check out the openSAP course and see if it helps to cut down on testing efforts and overall lead-time to go-live!
Thank you Gunther for your comment - you're absolutely right - it's not a 100% remedy but gives amazing results very quickly (weeks) compared to traditional E2E testing approach where you have to wait years for all systems to be set up in the test mode (what might mean it will not happen in reality)
Well written article on approaches to testing al large S/4 Hana transformation . RTA (Robotic Test Automation) addresses a major Regression Testing approaches but the above approach addresses a lot of factors in E2E Testing. I will definitely be looking into the Open sap course Avoid SAP S/4HANA Project Delays with Third-Party Systems Service Virtualization”
Thank you for your comment Subhodh 🙂
Another excellent blog Michal Krawczyk! I liked the video articulating the definition of end to end tests and how service virtualization makes it effective. Thanks a lot.
Hi Abhijeet Kulkarni
Indeed service virtualization is the effective enabler for many other types of testing as you've mentioned.
BR,
Michal Krawczyk
Nice writeup. The testing however needs to be upto the level of Value streams like OTC, P2P,P2B etc. It doesnt make sense if individual components and modules work but not together. Even the consultant knowledge must be focused on These value streams rather than specializations on specific module. It has been a pain for customer to hear every time from consultant on her scope limitations to modules while customer wants a cross-module process to work.
hi Raja Ivaturi,
"The testing however needs to be upto the level of Value streams like OTC, P2P,P2B etc." - let me ask you this question then - why not simulate 3rd party systems, EDI/B2B partners and only combine it with UI testing for SAP business process testing? (not E2E but SAP E2E) - do you see the difference in simplification? Still you can test whole Value streams like OTC, P2P,P2B but without any burden to go outside of SAP right?
Have a look at:
Week 3 - Unit 3: Service virtualization integration with UI testing tools
of https://open.sap.com/courses/iftt1-2-pc/
BR,
Michal
Dear Michal,
I agree with you about simulating 3rd party integrations for all initial testing. But my experience said, there was still good percentage of failure if we dont prove functioning of solution in the exact way the customer executes.
I give you examples. Our app was working well for one SA based top insurance organization. But failed when they tried to access SAP from their own portals. One supplier tried to work through Customer instance for one SRM scenario but failed.
This is because the simulations could never match the real experience. Having said this, I agree the failures are not because of approach you explained. It could be instead the failure of simulations.
To ensure 100% success in UAT, I always wanted the solution to work exactly the way customer and their stakeholders execute. Even if painful, it helped us go live with ZERO P0 and P1 issues.
In summary, my intention was not to deny your approach which is great by all means and simplifies the testing.
regards,
Raja
indeed - we need a well balanced mix to solve things in a proper and pragmatic way
BR,
Michal
I agree with you Michal about simulating 3rd party integrations for all initial integrated testing. As per my knowledge there was failure percentage of Test data, Environment setup, Roles & Security assignment. Any testing has to follow Roles & Security testing followed by Integrated environment setup and Test Data validation for successful testing.
Why Service Virtualization?
Service Virtualization replaces complex applications and dependencies in end-to-end transaction flows with simpler, virtualized services (“intelligent stubs”) which simulate the original application.
Thanks
Deb