Skip to Content
Technical Articles

Dear SAP Cloud Platform Integration, where is my CI/CD pipeline?

A few years ago, I posed a (rhetorical) question if we can approach integration development differently on SAP Cloud Platform Integration inย Dear SAP Cloud Integration, how do I unit test?. Unfortunately, nothing much has changed in that space, and most of what we still do today is inherited from the decade-old outdated approach used during SAP Process Integration days.

A discussion on LinkedIn came up recently about the availability of CI/CD pipeline or capability in CPI. Like many organisations these days, another one is trying to incorporate modern software development practices inline with Agile and/or DevOps principles.

The aim is simple – increase quality, increase automation, and decrease time to deploy.

In its current state, CPI provides remotely accessible APIs, and as shown in this SAP Cloud Platform Academy YouTube video, there is a possibility to implement some form of CI/CD automation with them. However, IMHO it is far from complete and many elements are still missing for a truly effective pipeline to be possible.

 

So here I am again – stating my wish list and hoping that Christmas will come early this time round ๐Ÿ™‚

 

Dear Santa, I would like to have …

To be able to implement Continuous Integration and Continuous Delivery/Deployment, it requires a paradigm shift of how we perform development and the toolset/capabilities required. Revolution (and not evolution) is the key!

 

Multi-branch Design Repository

Instead of the current centralised repository on the CPI tenant, design artifacts should be stored in a repository on a distributed version-management server with multi-branch capability. Repositories should have

  • A remote main branch – hosting the “golden” versions of the design artifacts
  • One to many remote development branches – hosting the in-transit changes to the design artifacts

 

Decentralised/Distributed Development

Development should be decentralised/distributed and changes are not directly done on the design artifact on the main branch. Developer works on the local copy of the development branches. Changes are pushed back to the remote development branches, where automated testing is triggered to verify the quality of the changes. Changes that pass automated testing will be merged into the main branch.

Decentralised approach was previously possible when developing on Eclipse, but developing on Web UI means creating/modifying design artifacts directly on the tenant. While some refer to CPI’s Web Application as “Web IDE” – it is incorrect in both terminology and feature. CPI Web UI is by no means an Integrated Development Environment in the same sense as SAP Web IDE, a service on the SAP Cloud Platform. The latter, SAP Web IDE is powered by Eclipse Che and provides a better browser-based development experience with common tools like auto-completion, syntax check, version control, as well as supporting decentralised development.

 

Unit Testing Framework

The underlying Apache Camel provides a unit testing framework to test Camel routes. Specifically, it also supports CPI generated Camel routes in OSGi Blueprint format via CamelBlueprintTestSupport class. However, there is no access for the developer to utilise the framework. Providing access to the built-in unit testing framework in Camel (whether directly or indirectly via a scripting or expression language) would allow the developer to write unit tests alongside the integration artifacts. A key feature of the unit testing framework should be the capability to mock endpoints in order to have deterministic unit tests.

 

Automated Testing

Automated testing is a key aspect of any Agile/DevOps practice. Upon committing or merging of a change, testing should be triggered automatically. The CI/CD pipeline should include automated testing capability for the following areas:-

  • Unit testing – with mocked endpoints
  • End-to-end testing – with real endpoints
  • Capability to trigger third-party testing tools – e.g. running test cases on int4 IFTT via an API call which allows full spectrum testing of the interface up to the business document validation on an SAP backend.

 

Automated Production Deployment

The holy grail for automation would be to implement Continuous Deployment which fully automates the build-to-deploy process up till production deployment. With this, changes can be deployed at any time without relying on manual and time-consuming release procedures.

 

 

Imagine … You May Say That I’m a Dreamer

Here’s how I would imagine the typical development workflow of an integration if we have the above in place.

  • Developer creates a development branch off the main branch
  • Developer works on local copy of the development branch – on a fully equipped IDE (browser-based or otherwise)
  • Developer creates unit test cases based on unit testing framework
  • Developer commits changes to remote development branch
  • Automated build process – Camel route(s) generated out of design artifacts and deployed to runtime of Development tenant
  • Automated unit testing in Development tenant
  • Automated merging of changes into main branch upon successful unit testing
  • Automated deployment to Quality tenant upon successful unit testing
  • Automated end-to-end testing (and optionally third-party) testing in Quality tenant
  • Automated deployment to Production tenant upon successful end-to-end testing

From the moment the developer commits the changes, the whole build-to-deploy process is fully automated.

 

More Work or Less?

Some might say I have wishful thinking. Others might argue “doesn’t this involve more work?”. With topics like citizen integrator trending in the integration space, one might be tempted to believe that integration development are (or should be) getting simpler. That might be true for data integration. However for process integration, IMHO integrations are getting more complex with the current trend of hybrid landscapes with multiple on-premise and SaaS solutions. Without a more efficient and automated approach to developing interfaces, it will take longer to deliver them and they will be more prone to errors. Whilst it may take a longer time to familiarise and set up at the beginning, but it pays back manifolds in the long run.

 

I would love to hear about your experiences, thoughts and opinions. Please feel free to comment below.

8 Comments
You must be Logged on to comment or reply to a post.
  • Eng,

    This is exactly what modern software development should have . And this is required by most of the clients and serious missing functionality with in sap integration space.ย  I wish dream come true soon .

    Regards

    Vijay

  • Please add the following to the list:

     

    1. Copy paste option for mappings and shapes;
    2. Rename option for scripts and mappings;
    3. Be able to use spaces in all shapes;
    4. Native JSON support;
    5. Be able to create message structures, rather than relying on external tools such as XML Spy to generate XSD’s for you;
    6. Keep my process line at the same position after I save an artifact;
    7. Be able to reprocess failed messages;
    8. Option to select and delete multiple shapes at one click;
    9. Pretty print for XML and JSON messages in trace mode;
    10. And of course: way fewer bugs in the GUI. Lots of times scrollbars are missing, properties are not visible, monitoring screen is completely emtpy or scrolling through the canvas is unresponsive.
    • Thanks Iddo for your comment.

       

      You have an interesting wish list and while I do not disagree that some of them might come in handy – IMHO those are at best evolutionary and will not contribute significant changes to the ability to delivery faster, better and easier.

      There is a need to think bigger in order to bring this to the next level.

      • True. Most of are merely my personal frustrations and are a huge benefit for the developer, while 4, 5 and 7 are must haves for any integration platform.

  • Hi Eng,

    As always some really good inputs.

    There is some APIs are missing like update iflow and list of packages in the CPI platform.

    In the Figaf tool we have been adding quite a few of the functions you describe. Sure it is a third party application you will need to buy. We have solved some of the challenges that you have with your pipeline.

     

    • Git: We can Take your full CPI repository and place it in git and then give different tooling to run unit test based on test case you have created. We do need to find some way to work with Git branches.
    • IDE we have created some opensource gradle plugins that allow you to develop in your IDE and then upload and deploy with just one click. Configuration templates is in the Git repository.
    • We can create unit scripts for your groovy code, so you can run messages simple.
    • We do have unit test and also a full mocking service of your iflows
    • We do have transports of individual transports and configurations on top of this.

     

     

     

  • Hi Eng,

    Thanks for your article and wishlist. I share it and therefore make use of the OData API within a small tool set.

    End of last year I wrote about it (https://blogs.sap.com/2019/12/24/continuous-cloud-integration-practical-usage-of-the-remote-data-apis-for-integration-flows/)

    Later, I made the source of the contribution available to the public. And @Daniel Graversen, even upload is available by using delete and create in a row.

    Regarding unit tests, we have a solution on-line that intervents Camel routes and asserts actual flow results using external expectings.

    I will write about that soon.

    Regards,

    Uwe

    • Hi Uwe

       

      Thank you for your comment. I came across your post when you first published it, but it wasn’t clear how it fits into a CI/CD pipeline. At that time there wasn’t any source code provided, so all I understood was there were some CLIs in part of a manual workflow.

      If you indeed have this as an automated pipeline from build-to-deploy (with unit testing as well), I would love to hear about it more ๐Ÿ™‚

       

      Best regards,

      Eng Swee