Technical Articles
Go blue/green with your Cloud Foundry app from WebIDE with Azure DevOps
Dear community,
My last post on release strategies (e.g. canary) covered a nice example on how to implement an integrated CI/CD pipeline with SAP WebIDE, Azure DevOps and Azure App Service for Containers. Today I’d like to focus on a CI/CD integration scenario for Cloud Foundry.
SAP offers a nice one-click build and deploy integration feature for SAP WebIDE. Unfortunately, this is falling short of many requirements developers and customers have regarding flexible deployments like blue/green and instant rollback mechanisms in production. A common scenario nowadays is also to be able to release to multiple cloud platforms (Azure, AWS, GCP) from one code base.
There are multiple tools out there, that get the job done. Popular choices are Jenkins, Project Piper (including Jenkins), GitHub Actions and Azure DevOps to name a few. I am going to focus on the latter today.
We ran an session on our YouTube channel about this post. So, if you are looking for some verbal context: here you go…
So, imagine you have an SAP Multi-Target Application (MTA) developed in SAP WebIDE, that you want to release to Cloud Foundry using blue/green deployment. I developed a fully working and publicly available example for the release process. Check the image below to get a feeling for the moving parts of the project that I am describing.
fig.1 Overview of all moving parts of today’s CI/CD undertaking
Short recap on Multi Target Applications: MTA is SAP’s answer to the challenge of moving multiple interconnected software components from one environment to another. SAP Cloud Platform (in our case using the Cloud Foundry environment) is smart enough to solve this software-component puzzle and automatically deploys every piece depending on the technology used, where it needs to be. Cloud Foundry expects the “puzzle” to be bundled in an MTA archive.
Let’s have a look how we can make Azure DevOps understand MTA projects and release to Cloud Foundry. But first things first. Here is how I setup my example project:
Prerequisites
- Create a Multi Target Application from template in SAP WebIDE (or Business Application Studio) and add an HTML5 module
- Try to build it in WebIDE wit Cloud MTA Build to make sure everything is working fine
- Upload your sample project to GitHub for version control and accessibility in Azure DevOps as well as continuous integration on every Git pull request or push. Azure DevOps will run the MTA build once you are all set.
- Create a Cloud Foundry environment on your SAP Cloud Platform account. You can choose between the three hyper scalers (AWS, Azure, GCP). I picked AWS for a multi-cloud scenario because I am releasing from Azure DevOps
- Have a CF space on your subaccount available and be aware about the available quotas, your CF API endpoint and respective org name because you will need them to login from Azure DevOps.
- (Optional) Install CF CLI locally for troubleshooting purposes. That way you can check if you have your login details right for example: cf login -a https://api.cf.eu10.hana.ondemand.com/ -u <S-User> -p <your password> -o <org name> -s <space name>
- Create a project in DevOps (e.g. search DevOps in Azure portal)
Get right into it
Now it is time to configure our build pipeline in Azure DevOps so we can build our MTA project, which we develop in SAP WebIDE or any other IDE, that understands the languages used in your MTA (e.g. Visual Studio Code, Eclipse etc.).
fig.2 Screenshot of build pipe on Az DevOps
My build pipe runs on an ubuntu agent, which downloads SAP’s Cloud MTA build tool and executes the build immediately. After that I copy the resulting MTAR-file from the build directory to the staging directory, so that my release pipe can pick it up next.
Using the Cloud MTA build tool documentation and local testing with Powershell I was able to determine the right path for Azure DevOps: $(agent.builddirectory)/s/mta_archives.
Note on the side: I configured continuous integration from GitHub through the Triggers tab of the build pipe shown above.
wget https://github.com/SAP/cloud-mta-build-tool/releases/download/v1.0.5/cloud-mta-build-tool_1.0.5_Linux_amd64.tar.gz
tar xvzf cloud-mta-build-tool_1.0.5_Linux_amd64.tar.gz
sudo cp mbt /usr/local/bin/
npm config set @sap:registry https://npm.sap.com
mbt build
fig.3 Cloud MTA Build setup on Az DevOps
So far so good. We have a MTAR file, which Cloud Foundry can understand. What’s next? Remember, we wanted to do blue/green deployments.
Closer look at the release pipe
In general, this means you have two instances of your app running in parallel – typically, different versions of it. One of them is live (v1) and the other is idle (v2). Once you are happy with the new version you can swap the routes, so that v2 becomes live, without any disruption to the end users.
fig.4 Blue green deployment
However, the command of Cloud Foundry CLI deletes the v1 deployment slot and idle routes once the swap finishes. You can find further info on the matter on the SAP docs here.
I configured a 2-stage deployment with CF CLI and modelled my release pipe in Azure DevOps accordingly (idle vs. live). The review step contains a manual intervention step. That way you can test the new version before performing the swap and run a rollback in case of rejection of the new release.
fig.5 Az DevOps release pipe with approval setup
Both release tasks in idle and live stage are similar and differ only in configuration parameter for the cf commands to initiate the blue-green process and to conclude it.
fig.6 Az DevOps release pipe dev stage setup
First, I install a recent version of the CF CLI on my agent to be and secondly execute the following set of commands:
cf install-plugin multiapps -f
$mtar = Get-ChildItem -recurse *.mtar | Select FullName
$target = $mtar.FullName
echo "$target"
cf login -a https://api.cf.eu10.hana.ondemand.com/ -u $ENV:MYSUSER -p $ENV:MYPASS -o $ENV:MYACCOUNT -s dev
cf deploy $target --strategy blue-green
fig.7 Powershell script on dev stage in release pipe in Azure DevOps
Let’s walk through it.
- Install a plugin for the CF CLI, that allows MTA deployments
- Retrieve the file path to the MTAR-file, that I want to deploy. Note that I set the working directory for the Powershell task to the drop folder of the release pipe input.
- Login with the CF API using my S-User, password, org and space name to be able to work with my account. Note that I defined the parameters as secrets on the release pipeline and mapped them into the script using environment variables. You can learn more about that here.
- Do the actual deployment with the pre-bundled plugin for blue/green deployments.
After the idle stage of my release pipeline finishes you will see two instances on SAP Cloud Platform. That represents the intermediate state of the blue/green deployment as described in figure 4.
fig.8 Release process intermediate state before approval/rejection
After all that, you are one approval away from releasing a new version of your app to production with state of the art CI/CD from SAP WebIDE/Business Application Studio to Cloud Foundry with zero-downtime (see upper part of figure 8).
The prod stage runs a slightly different script. We rely on “mta-ops” to be able to conclude the process after we finish our testing on the idle route of our app.
Since the process id is specific to my app it is reasonably safe to extract the id using a greedy regex. I would prefer json but it seems not to be available on the CF CLI yet.
cf install-plugin multiapps -f
cf login -a https://api.cf.us10.hana.ondemand.com/ -u $ENV:MYSUSER -p $ENV:MYPASS -o $ENV:MYACCOUNT -s dev
$processId = cf mta-ops -mta MTA-Azure-CF-Demo
$match = [regex]::Match($processId, '([a-z0-9]{8}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{12})')
if($match.Success){
Write-Host "my process id: $match.Value"
cf deploy -i $match.Value -a resume
}
else{
Write-Host "No activeProcessId, check variable!"
}
Fig.10 prod stage CF commands
Using the process id we can finish the blue/green process with the “resume” option. The abort process “lives” in the review process stage. It only triggers when the Manual Intervention task fails. So, on “resume” or approve of the current idle version of the app the process continues to the live stage of the DevOps pipeline. In case of “reject” it continues with below job “Abort Blue Green”. The app name could be abstracted with a variable if needed or extracted via other CF CLI requests. Since we know the app names on the pipeline I kept it simple.
The abort command currently does not clean up the routes, therefore we delete them intentionally. The following blue/green processes and most important the live app is not impacted in case you decide not to add the delete and rename command.
cf install-plugin multiapps -f
cf login -a https://api.cf.eu20.hana.ondemand.com/ -u $ENV:MYSUSER -p $ENV:MYPASS -o $ENV:MYACCOUNT -s dev
$processId = cf mta-ops -mta MTA-Azure-CF-Demo
$match = [regex]::Match($processId, '([a-z0-9]{8}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{12})')
if($match.Success){
Write-Host "my process id: $match.Value"
cf deploy -i $match.Value -a abort
cf delete -r -f ui-idle
cf rename ui-live ui
}
else{
Write-Host "No activeProcessId, check variable!"
}
fig.11 Screenshot from abort blue/green job and code snippet
There are various ways to achieve the pipeline behavior needed for blue green. Azure DevOps provides pre- and post-deployment conditions to trigger interaction based on a pipeline stage. See Quality gates for interaction with Azure Policies, REST APIs, Azure Monitor metrics or open work items from your agile process in Azure Boards.
Check the SAP docs for more details on the resume/abort option.
Fig.12 final state after blue/green process
Great, the intermediate state of idle/live has been cleaned up by CloudFoundry under the hood for me. Only the new productive version is left.
Analysing idle version with tooling is key to further enhance your CD practice
The described process so far focusses on manual steps and an “active” decision by someone within Azure DevOps to resume the process. But how do you test the idle version?
- Is the new ui component present?
- Can I click buttons and receive expected results?
- Do I need to avoid controls that alter data base entries?
- What about dummy objects and users for my testers in production?
Here is an example from one of the largest streaming providers on the planet. They rely on metric comparison to create a visual for the engineers to judge upon.
source: netflixtechblog.com
So, we would need to analyse the behavior of the CF app through BTP means like the BTP Cockpit (simple visual memory, CPU and disc consumption comparison) or incorporate a logging framework like SAP Application Logging Service or Azure Application Insights.
Wait, blue green is nice but what about incompatible changes on my data structures?
Above descriptions on blue/green deployments work without planning when it comes to adding ui components, services or data base fields. But what if you need to incorporate a breaking change?
Let’s have a look at the following data structures.
|
|
The left one is currently live, because the app didn’t anticipate the need to deliver to a different address than the purchaser’s. The structure on the right introduces the desired separation.
Effectively, you need to run the app and data base schema of both implementations in parallel until the blue/green process finishes. The solution to this problem is widely known as the “Expand and Contract” or “Parallel Change” pattern.
source: medium.com
The speed of your change from left to right depends on how often you release new versions and how quick consuming clients get updated to the new implementation, so that you can finally delete obsolete interfaces, objects and data entries.
Assuming 4 releases to finish the introduction of an incompatible change with a typical 4 week Scrum cycle with a release once a month, you arrive at 4 months till the process completes.
With mobile apps this can take particularly long, because end-users might be reluctant to run updates. Some apps even throw you out to force updates upon you once such breaking changes have been introduced.
Check out this article for a nice detailed description of the different phases. I borrowed the delivery address example from there too.
Hana’s XS has a similar built-in implementation for Zero-Downtime Maintenance, that might reduce your efforts if your app stack runs on it.
Final Words
There are several ways to establish sophisticated and flexible release processes for your Multi Target Applications. I showed you one easy example with GitHub, Azure DevOps Pipelines and the blue-green option of the CF API CLI that gets you a great deal of new possibilities.
Incompatible changes to your data structures increase the complexity of your zero-downtime releases with blue/green. Maybe take down the system for the change? 😉 naa, we have come this far with pre-bundled tooling at your hands to implement a cloud native strategy. But consider release frequency to check if it is worth the effort. Once a month seems like the minimum.
Check out my other blog post on different release strategies with UI5 and Azure DevOps. You can get even more fancy than blue/green 😉
Find my GitHub repository here.
Public Azure DevOps project here.
As always feel free to leave or ask lots of follow-up questions.
Best Regards
Martin
Opinions expressed are solely my own and do not express the views or opinions of my employer.
Hi Martin,
Thank you for explanation, It is really helpful.
However I have one question on the
I am using multistage yaml pipeline in Azure DevOps and I would like to excute resume once it is approved by someone in team.
https://help.sap.com/viewer/65de2977205c403bbc107264b8eccf4b/Cloud/en-US/764308c52e68488dac848bae93e9137b.html
How can I read the process id in Azure DevOps and pass it to resume. so that once its is approved, it complete automatically and the latest version become active end points.
Thanks,
Rajesh
Hi Rajesh,
that is going to be a challenge. I would look into pipeline variables first https://docs.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch
Another option might be to call a key value store with a seperate task and retrieve in the next pipeline stage.
If you check my example I bypassed that id passing problem by running bg-deploy first and in next stage
KR
Martin
Hi Martin,
Thanks for your response. Just sharing my thought on the points you made.
#1 I have tried the Pipeline variable approach. This approach has two problem, first the value of variables is read at the time of pipeline triggered, so if i change the variables value while pipeline is running, it doesn't read the new value set. The second problem, it will add a manual intervention. someone has to go copy process id and set in pipeline variable.
#2 Setting process Id in key/value store at run time, will require reading process id. which is the original problem.
#3 If I use --no-confirm, It doesn't give opportunity to verify the newly deployed instance (may be blue or green) working fine or not. so in my understanding it wont be blue/green anymore. it is same as, just simply deploy new version.
Thanks,
Rajesh
Hi Rajesh,
I see your point. For my demo I took an easy route for sure, which was good enough. The parameter –no-confirm also makes sure the release agent can run unattended.
How about providing your own release agent (instead of managed) so you are able to keep all info from the process? https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops&tabs=browser#install
That should allow you to use CLI commands you are looking for.
Let me know how it goes.
KR
Martin
Hi Rajesh Bhadana,
I updated the CF CLI commands to work with the "-resume" option. Testing the new instance was possible before too, but as you mentioned the "force operation" didn't clean up the blue/green process as it was intended. With the CLI command "cf mta-ops -mta <your MTA id>" you get the process id during runtime. That erases the need to remember it somewhere. However the pipeline variable solution works for subsequent runs too.
You can have a look at the public repos to see the full version until I publish the blog update.
Let me know if that suits your requirement.
KR
Martin
Hi Martin
First of all thanks for this post, this helped me a lot as a beginner in SAP. Its helped in understanding SAP build process.
Secondly, I ran into a issue in release pipeline when any other deployment process is in progress.You can see below image for details.
My requirement is if any deployment is in progress I want to abort it automatically through pipeline.
Can you please help on this.
Thanks & Regards
Abhimanyu Singh
Hi Abhimanyu,
You will need to look into the CF CLI docs to check on status info and make use of that. Max has a good post to get you started: https://blogs.sap.com/2020/02/04/maxs-adventure-in-sap-cloud-platform-pimp-my-cloud-foundry-cli/
Also you might the SAP help on "blue green" for CF helpful: https://help.sap.com/viewer/65de2977205c403bbc107264b8eccf4b/Cloud/en-US/7c83810c31d842938cbc39c135a2d99f.html
KR
Martin
Hi Martin
Thanks for the reply, this helped.
Thanks
Abhimanyu Singh
Hi Pankraz,
I have followed your blog and created exactly same CI and release pipeline and it is running properly with SAP Business Studio.
After dev release stage completion, I got two URLs(live and idle) but both are updated version of app.
I was expecting live URL should be of my old deployment (but got overwrite) and idle of my updated app with changes.
Could you please let me know, where have I done wrong or this is expected behavior?
Hi Pankaj,
the whole purpose of blue/green deployment is zero downtime deployment and the possibility to rollback a new release before making it live. So your assumption should be met indeed. Let's see why it doesn't behave correctly.
Did you have a live version before you triggered blue/green first time? Also I had some refresh issues while testing idle with small ui changes that were cached. How about making a logic change to test if that is the case?
I just verified the expected behavior on my end successfully again. My idle app shows the new change (additional button on header) while the live one is still in the old productive state.
KR
Martin