Go blue/green with your Cloud Foundry app from WebIDE with Azure DevOps

Dear community,

My last post on release strategies (e.g. canary) covered a nice example on how to implement an integrated CI/CD pipeline with SAP WebIDE, Azure DevOps and Azure App Service for Containers. Today I’d like to focus on a CI/CD integration scenario for Cloud Foundry.

SAP offers a nice one-click build and deploy integration feature for SAP WebIDE. Unfortunately, this is falling short of many requirements developers and customers have regarding flexible deployments like blue/green and instant rollback mechanisms in production. A common scenario nowadays is also to be able to release to multiple cloud platforms (Azure, AWS, GCP) from one code base.

There are multiple tools out there, that get the job done. Popular choices are Jenkins, Project Piper (including Jenkins), GitHub Actions and Azure DevOps to name a few. I am going to focus on the latter today.

We ran an session on our YouTube channel about this post. So, if you are looking for some verbal context: here you go…

So, imagine you have an SAP Multi-Target Application (MTA) developed in SAP WebIDE, that you want to release to Cloud Foundry using blue/green deployment. I developed a fully working and publicly available example for the release process. Check the image below to get a feeling for the moving parts of the project that I am describing.

fig.1 Overview of all moving parts of today’s CI/CD undertaking

Short recap on Multi Target Applications: MTA is SAP’s answer to the challenge of moving multiple interconnected software components from one environment to another. SAP Cloud Platform (in our case using the Cloud Foundry environment) is smart enough to solve this software-component puzzle and automatically deploys every piece depending on the technology used, where it needs to be. Cloud Foundry expects the “puzzle” to be bundled in an MTA archive.

Let’s have a look how we can make Azure DevOps understand MTA projects and release to Cloud Foundry. But first things first. Here is how I setup my example project:

Prerequisites

• Create a Multi Target Application from template in SAP WebIDE (or Business Application Studio) and add an HTML5 module
• Try to build it in WebIDE wit Cloud MTA Build to make sure everything is working fine
• Upload your sample project to GitHub for version control and accessibility in Azure DevOps as well as continuous integration on every Git pull request or push. Azure DevOps will run the MTA build once you are all set.
• Create a Cloud Foundry environment on your SAP Cloud Platform account. You can choose between the three hyper scalers (AWS, Azure, GCP). I picked AWS for a multi-cloud scenario because I am releasing from Azure DevOps
• Have a CF space on your subaccount available and be aware about the available quotas, your CF API endpoint and respective org name because you will need them to login from Azure DevOps.
• (Optional) Install CF CLI locally for troubleshooting purposes. That way you can check if you have your login details right for example: cf login -a https://api.cf.eu10.hana.ondemand.com/ -u <S-User> -p <your password> -o <org name> -s <space name>
• Create a project in DevOps (e.g. search DevOps in Azure portal)

Get right into it

Now it is time to configure our build pipeline in Azure DevOps so we can build our MTA project, which we develop in SAP WebIDE or any other IDE, that understands the languages used in your MTA (e.g. Visual Studio Code, Eclipse etc.).

fig.2 Screenshot of build pipe on Az DevOps

My build pipe runs on an ubuntu agent, which downloads SAP’s Cloud MTA build tool and executes the build immediately. After that I copy the resulting MTAR-file from the build directory to the staging directory, so that my release pipe can pick it up next.

Using the Cloud MTA build tool documentation and local testing with Powershell I was able to determine the right path for Azure DevOps: $(agent.builddirectory)/s/mta_archives.

Note on the side: I configured continuous integration from GitHub through the Triggers tab of the build pipe shown above.

wget https://github.com/SAP/cloud-mta-build-tool/releases/download/v1.0.5/cloud-mta-build-tool_1.0.5_Linux_amd64.tar.gz
tar xvzf cloud-mta-build-tool_1.0.5_Linux_amd64.tar.gz
sudo cp mbt /usr/local/bin/
npm config set @sap:registry https://npm.sap.com
mbt build

fig.3 Cloud MTA Build setup on Az DevOps

So far so good. We have a MTAR file, which Cloud Foundry can understand. What’s next? Remember, we wanted to do blue/green deployments.

Closer look at the release pipe

In general, this means you have two instances of your app running in parallel – typically, different versions of it. One of them is live (v1) and the other is idle (v2). Once you are happy with the new version you can swap the routes, so that v2 becomes live, without any disruption to the end users.

fig.4 Blue green deployment

However, the command of Cloud Foundry CLI deletes the v1 deployment slot and idle routes once the swap finishes. You can find further info on the matter on the SAP docs here.

I configured a 2-stage deployment with CF CLI and modelled my release pipe in Azure DevOps accordingly (idle vs. live). The review step contains a manual intervention step. That way you can test the new version before performing the swap and run a rollback in case of rejection of the new release.

fig.5 Az DevOps release pipe with approval setup

Both release tasks in idle and live stage are similar and differ only in configuration parameter for the cf commands to initiate the blue-green process and to conclude it.

fig.6 Az DevOps release pipe dev stage setup

First, I install a recent version of the CF CLI on my agent to be and secondly execute the following set of commands:

cf install-plugin multiapps -f
$mtar = Get-ChildItem -recurse *.mtar | Select FullName
$target = $mtar.FullName
echo "$target"
cf login -a https://api.cf.eu10.hana.ondemand.com/ -u $ENV:MYSUSER -p $ENV:MYPASS -o $ENV:MYACCOUNT -s dev
cf deploy $target --strategy blue-green

fig.7 Powershell script on dev stage in release pipe in Azure DevOps

Let’s walk through it.

1. Install a plugin for the CF CLI, that allows MTA deployments
2. Retrieve the file path to the MTAR-file, that I want to deploy. Note that I set the working directory for the Powershell task to the drop folder of the release pipe input.
3. Login with the CF API using my S-User, password, org and space name to be able to work with my account. Note that I defined the parameters as secrets on the release pipeline and mapped them into the script using environment variables. You can learn more about that here.
4. Do the actual deployment with the pre-bundled plugin for blue/green deployments.

After the idle stage of my release pipeline finishes you will see two instances on SAP Cloud Platform. That represents the intermediate state of the blue/green deployment as described in figure 4.

fig.8 Release process intermediate state before approval/rejection

After all that, you are one approval away from releasing a new version of your app to production with state of the art CI/CD from SAP WebIDE/Business Application Studio to Cloud Foundry with zero-downtime (see upper part of figure 8).

The prod stage runs a slightly different script. We rely on “mta-ops” to be able to conclude the process after we finish our testing on the idle route of our app.

fig.9 Screenshot of CF CLI command output, that is reused in Azure pipelines

Since the process id is specific to my app it is reasonably safe to extract the id using a greedy regex. I would prefer json but it seems not to be available on the CF CLI yet.

cf install-plugin multiapps -f
cf login -a https://api.cf.us10.hana.ondemand.com/ -u $ENV:MYSUSER -p $ENV:MYPASS -o $ENV:MYACCOUNT -s dev
$processId = cf mta-ops -mta MTA-Azure-CF-Demo

$match = [regex]::Match($processId, '([a-z0-9]{8}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{12})')

if($match.Success){
    Write-Host "my process id: $match.Value"
    cf deploy -i $match.Value -a resume
} 
else{
    Write-Host "No activeProcessId, check variable!"
}

Fig.10 prod stage CF commands

Using the process id we can finish the blue/green process with the “resume” option. The abort process “lives” in the review process stage. It only triggers when the Manual Intervention task fails. So, on “resume” or approve of the current idle version of the app the process continues to the live stage of the DevOps pipeline. In case of “reject” it continues with below job “Abort Blue Green”. The app name could be abstracted with a variable if needed or extracted via other CF CLI requests. Since we know the app names on the pipeline I kept it simple.

The abort command currently does not clean up the routes, therefore we delete them intentionally. The following blue/green processes and most important the live app is not impacted in case you decide not to add the delete and rename command.

cf install-plugin multiapps -f
cf login -a https://api.cf.eu20.hana.ondemand.com/ -u $ENV:MYSUSER -p $ENV:MYPASS -o $ENV:MYACCOUNT -s dev
$processId = cf mta-ops -mta MTA-Azure-CF-Demo

$match = [regex]::Match($processId, '([a-z0-9]{8}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{4}-[a-z0-9]{12})')

if($match.Success){
    Write-Host "my process id: $match.Value"
    cf deploy -i $match.Value -a abort
    cf delete -r -f ui-idle
    cf rename ui-live ui
} 
else{
    Write-Host "No activeProcessId, check variable!"
}

fig.11 Screenshot from abort blue/green job and code snippet

There are various ways to achieve the pipeline behavior needed for blue green. Azure DevOps provides pre- and post-deployment conditions to trigger interaction based on a pipeline stage. See Quality gates for interaction with Azure Policies, REST APIs, Azure Monitor metrics or open work items from your agile process in Azure Boards.

Check the SAP docs for more details on the resume/abort option.

Fig.12 final state after blue/green process

Great, the intermediate state of idle/live has been cleaned up by CloudFoundry under the hood for me. Only the new productive version is left.

Analysing idle version with tooling is key to further enhance your CD practice

The described process so far focusses on manual steps and an “active” decision by someone within Azure DevOps to resume the process. But how do you test the idle version?

• Is the new ui component present?
• Can I click buttons and receive expected results?
• Do I need to avoid controls that alter data base entries?
• What about dummy objects and users for my testers in production?

Here is an example from one of the largest streaming providers on the planet. They rely on metric comparison to create a visual for the engineers to judge upon.

source: netflixtechblog.com

So, we would need to analyse the behavior of the CF app through BTP means like the BTP Cockpit (simple visual memory, CPU and disc consumption comparison) or incorporate a logging framework like SAP Application Logging Service or Azure Application Insights.

Wait, blue green is nice but what about incompatible changes on my data structures?

Above descriptions on blue/green deployments work without planning when it comes to adding ui components, services or data base fields. But what if you need to incorporate a breaking change?

Let’s have a look at the following data structures.

public class Order{
    public int OrderNumber {get;set}
    public String GetAddress {get;set}
}

public class Order{
    public int OrderNumber {get;set}
    public String BillingAddress {get;set}
    public String DeliveryAddress {get;set}
}

The left one is currently live, because the app didn’t anticipate the need to deliver to a different address than the purchaser’s. The structure on the right introduces the desired separation.

Effectively, you need to run the app and data base schema of both implementations in parallel until the blue/green process finishes. The solution to this problem is widely known as the “Expand and Contract” or “Parallel Change” pattern.

source: medium.com

The speed of your change from left to right depends on how often you release new versions and how quick consuming clients get updated to the new implementation, so that you can finally delete obsolete interfaces, objects and data entries.

Assuming 4 releases to finish the introduction of an incompatible change with a typical 4 week Scrum cycle with a release once a month, you arrive at 4 months till the process completes.

With mobile apps this can take particularly long, because end-users might be reluctant to run updates. Some apps even throw you out to force updates upon you once such breaking changes have been introduced.

Check out this article for a nice detailed description of the different phases. I borrowed the delivery address example from there too.

Hana’s XS has a similar built-in implementation for Zero-Downtime Maintenance, that might reduce your efforts if your app stack runs on it.

Final Words

There are several ways to establish sophisticated and flexible release processes for your Multi Target Applications. I showed you one easy example with GitHub, Azure DevOps Pipelines and the blue-green option of the CF API CLI that gets you a great deal of new possibilities.

Incompatible changes to your data structures increase the complexity of your zero-downtime releases with blue/green. Maybe take down the system for the change? 😉 naa, we have come this far with pre-bundled tooling at your hands to implement a cloud native strategy. But consider release frequency to check if it is worth the effort. Once a month seems like the minimum.

Check out my other blog post on different release strategies with UI5 and Azure DevOps. You can get even more fancy than blue/green 😉

Find my GitHub repository here.

Public Azure DevOps project here.

As always feel free to leave or ask lots of follow-up questions.

Best Regards

Martin

Opinions expressed are solely my own and do not express the views or opinions of my employer.