A few years ago, I posed a (rhetorical) question if we can approach integration development differently on SAP Cloud Platform Integration in Dear SAP Cloud Integration, how do I unit test?. Unfortunately, nothing much has changed in that space, and most of what we still do today is inherited from the decade-old outdated approach used during SAP Process Integration days.
A discussion on LinkedIn came up recently about the availability of CI/CD pipeline or capability in CPI. Like many organisations these days, another one is trying to incorporate modern software development practices inline with Agile and/or DevOps principles.
The aim is simple - increase quality, increase automation, and decrease time to deploy.
In its current state, CPI provides remotely accessible APIs, and as shown in this SAP Cloud Platform Academy YouTube video, there is a possibility to implement some form of CI/CD automation with them. However, IMHO it is far from complete and many elements are still missing for a truly effective pipeline to be possible.
So here I am again - stating my wish list and hoping that Christmas will come early this time round 🙂
To be able to implement Continuous Integration and Continuous Delivery/Deployment, it requires a paradigm shift of how we perform development and the toolset/capabilities required. Revolution (and not evolution) is the key!
Instead of the current centralised repository on the CPI tenant, design artifacts should be stored in a repository on a distributed version-management server with multi-branch capability. Repositories should have
Development should be decentralised/distributed and changes are not directly done on the design artifact on the main branch. Developer works on the local copy of the development branches. Changes are pushed back to the remote development branches, where automated testing is triggered to verify the quality of the changes. Changes that pass automated testing will be merged into the main branch.
Decentralised approach was previously possible when developing on Eclipse, but developing on Web UI means creating/modifying design artifacts directly on the tenant. While some refer to CPI's Web Application as "Web IDE" - it is incorrect in both terminology and feature. CPI Web UI is by no means an Integrated Development Environment in the same sense as SAP Web IDE, a service on the SAP Cloud Platform. The latter, SAP Web IDE is powered by Eclipse Che and provides a better browser-based development experience with common tools like auto-completion, syntax check, version control, as well as supporting decentralised development.
The underlying Apache Camel provides a unit testing framework to test Camel routes. Specifically, it also supports CPI generated Camel routes in OSGi Blueprint format via CamelBlueprintTestSupport class. However, there is no access for the developer to utilise the framework. Providing access to the built-in unit testing framework in Camel (whether directly or indirectly via a scripting or expression language) would allow the developer to write unit tests alongside the integration artifacts. A key feature of the unit testing framework should be the capability to mock endpoints in order to have deterministic unit tests.
Automated testing is a key aspect of any Agile/DevOps practice. Upon committing or merging of a change, testing should be triggered automatically. The CI/CD pipeline should include automated testing capability for the following areas:-
The holy grail for automation would be to implement Continuous Deployment which fully automates the build-to-deploy process up till production deployment. With this, changes can be deployed at any time without relying on manual and time-consuming release procedures.
Here's how I would imagine the typical development workflow of an integration if we have the above in place.
From the moment the developer commits the changes, the whole build-to-deploy process is fully automated.
Some might say I have wishful thinking. Others might argue "doesn't this involve more work?". With topics like citizen integrator trending in the integration space, one might be tempted to believe that integration development are (or should be) getting simpler. That might be true for data integration. However for process integration, IMHO integrations are getting more complex with the current trend of hybrid landscapes with multiple on-premise and SaaS solutions. Without a more efficient and automated approach to developing interfaces, it will take longer to deliver them and they will be more prone to errors. Whilst it may take a longer time to familiarise and set up at the beginning, but it pays back manifolds in the long run.
I would love to hear about your experiences, thoughts and opinions. Please feel free to comment below.
A discussion on LinkedIn came up recently about the availability of CI/CD pipeline or capability in CPI. Like many organisations these days, another one is trying to incorporate modern software development practices inline with Agile and/or DevOps principles.
The aim is simple - increase quality, increase automation, and decrease time to deploy.
In its current state, CPI provides remotely accessible APIs, and as shown in this SAP Cloud Platform Academy YouTube video, there is a possibility to implement some form of CI/CD automation with them. However, IMHO it is far from complete and many elements are still missing for a truly effective pipeline to be possible.
So here I am again - stating my wish list and hoping that Christmas will come early this time round 🙂
Dear Santa, I would like to have ...
To be able to implement Continuous Integration and Continuous Delivery/Deployment, it requires a paradigm shift of how we perform development and the toolset/capabilities required. Revolution (and not evolution) is the key!
Multi-branch Design Repository
Instead of the current centralised repository on the CPI tenant, design artifacts should be stored in a repository on a distributed version-management server with multi-branch capability. Repositories should have
- A remote main branch - hosting the "golden" versions of the design artifacts
- One to many remote development branches - hosting the in-transit changes to the design artifacts
Decentralised/Distributed Development
Development should be decentralised/distributed and changes are not directly done on the design artifact on the main branch. Developer works on the local copy of the development branches. Changes are pushed back to the remote development branches, where automated testing is triggered to verify the quality of the changes. Changes that pass automated testing will be merged into the main branch.
Decentralised approach was previously possible when developing on Eclipse, but developing on Web UI means creating/modifying design artifacts directly on the tenant. While some refer to CPI's Web Application as "Web IDE" - it is incorrect in both terminology and feature. CPI Web UI is by no means an Integrated Development Environment in the same sense as SAP Web IDE, a service on the SAP Cloud Platform. The latter, SAP Web IDE is powered by Eclipse Che and provides a better browser-based development experience with common tools like auto-completion, syntax check, version control, as well as supporting decentralised development.
Unit Testing Framework
The underlying Apache Camel provides a unit testing framework to test Camel routes. Specifically, it also supports CPI generated Camel routes in OSGi Blueprint format via CamelBlueprintTestSupport class. However, there is no access for the developer to utilise the framework. Providing access to the built-in unit testing framework in Camel (whether directly or indirectly via a scripting or expression language) would allow the developer to write unit tests alongside the integration artifacts. A key feature of the unit testing framework should be the capability to mock endpoints in order to have deterministic unit tests.
Automated Testing
Automated testing is a key aspect of any Agile/DevOps practice. Upon committing or merging of a change, testing should be triggered automatically. The CI/CD pipeline should include automated testing capability for the following areas:-
- Unit testing - with mocked endpoints
- End-to-end testing - with real endpoints
- Capability to trigger third-party testing tools - e.g. running test cases on int4 IFTT via an API call which allows full spectrum testing of the interface up to the business document validation on an SAP backend.
Automated Production Deployment
The holy grail for automation would be to implement Continuous Deployment which fully automates the build-to-deploy process up till production deployment. With this, changes can be deployed at any time without relying on manual and time-consuming release procedures.
Imagine ... You May Say That I'm a Dreamer
Here's how I would imagine the typical development workflow of an integration if we have the above in place.
- Developer creates a development branch off the main branch
- Developer works on local copy of the development branch - on a fully equipped IDE (browser-based or otherwise)
- Developer creates unit test cases based on unit testing framework
- Developer commits changes to remote development branch
- Automated build process - Camel route(s) generated out of design artifacts and deployed to runtime of Development tenant
- Automated unit testing in Development tenant
- Automated merging of changes into main branch upon successful unit testing
- Automated deployment to Quality tenant upon successful unit testing
- Automated end-to-end testing (and optionally third-party) testing in Quality tenant
- Automated deployment to Production tenant upon successful end-to-end testing
From the moment the developer commits the changes, the whole build-to-deploy process is fully automated.
More Work or Less?
Some might say I have wishful thinking. Others might argue "doesn't this involve more work?". With topics like citizen integrator trending in the integration space, one might be tempted to believe that integration development are (or should be) getting simpler. That might be true for data integration. However for process integration, IMHO integrations are getting more complex with the current trend of hybrid landscapes with multiple on-premise and SaaS solutions. Without a more efficient and automated approach to developing interfaces, it will take longer to deliver them and they will be more prone to errors. Whilst it may take a longer time to familiarise and set up at the beginning, but it pays back manifolds in the long run.
I would love to hear about your experiences, thoughts and opinions. Please feel free to comment below.
- SAP Managed Tags:
8 Comments
