Technical Articles
Deprecation of Cloud Foundry Stack cflinuxfs3 and Migration to cflinuxfs4
Overview
SAP BTP, Cloud Foundry runtime uses stacks, a prebuilt root file system that in combination with application code and buildpacks is the base to build the application container. The stack in the SAP BTP, Cloud Foundry environment is based on a Linux system and is called cflinuxfs<x>.
Stacks receive regular updates to address security issues and vulnerabilities, which your applications can pick up through new releases and version updates of the SAP BTP, Cloud Foundry Runtime. To get information about version updates and other Cloud Foundry related topics, see What’s New for SAP Business Technology Platform.
Currently in the SAP BTP, Cloud Foundry environment, all applications run on a stack called cflinuxfs3, which is based on Ubuntu Bionic Beaver 18.04. This stack was deprecated in December 2022 and is planned to be removed from the SAP BTP, Cloud Foundry environment in May 2023 after support for Ubuntu Bionic Beaver 18.04 has ended.
The successor stack cflinuxfs4, based on Ubuntu Jammy Jellyfish 22.04, has been available on an opt-in basis since December 2022 and is planned to become the default stack in March 2023.
This change affects anyone who is running buildpack-based applications in the SAP BTP, Cloud Foundry environment.
To help you understand what is going to happen, we’ve created a simplified timeline of the planned changes:
Cloud Foundry Stack Migration Timeline
What Happens When cflinuxfs4 Becomes the Default Stack?
Once cflinuxfs4 is designated as the default stack, all new applications will use this stack automatically. Existing applications running in the SAP BTP, Cloud Foundry environment won’t be affected by this change, but it is recommended that you update them to use the new stack by following the guide below, as the old stack will be removed in May 2023 and your app can stop working in case it is not compatible with the cflinuxfs4 stack.
To start using cflinuxfs4 with existing applications, you can push the application and designate the stack manually using:
cf push <APP_NAME> -s cflinuxfs4
If you’re using the blue-green deployment method to push your applications, they are automatically deployed on cflinuxfs4 when it is declared the default stack. This only works if you delete the unused blue or green applications. For more information on blue-green deployment, see https://docs.cloudfoundry.org/devguide/deploy-apps/blue-green.html.
You can continue to use cflinuxfs3 for a transitionary period. This means you can also manually switch back to cflinuxfs3 if you notice problems using the new stack by using the “-s” command when pushing your application:
cf push <APP_NAME> -s cflinuxfs3
cflinuxfs3 will be permanently deleted in May 2023. Afterwards it is no longer possible to manually switch back to cflinuxfs3. For more information on stacks, see https://docs.cloudfoundry.org/devguide/deploy-apps/stacks.html#available-stacks.
Risks and Consequences
If the applications were running, the restart causes a short downtime. However, if your applications are incompatible with the cflinuxfs4 stack, it is possible that they stop working and cannot be restarted.
To maintain functionality and compliance with security standards in the SAP BTP, Cloud Foundry environment, you must migrate your applications to cflinuxfs4. We recommend that you start testing your applications with the new stack as soon as possible to be able to fix potential issues. Only applications that have been migrated to cflinuxfs4 before the force-migration will not suffer from downtimes.
To assist you, we have compiled a short guide that walks you through the required migration steps.
Migrate to the cflinuxfs4 Stack with the CF CLI
Prerequisite:
Make sure you are using a recent version of the Cloud Foundry command line interface (CF CLI). We recommend using version 8. For information about the newest version, see https://github.com/cloudfoundry/cli/releases.
1. Determine Your Current Stack
If you are unsure which stack your applications are currently running on, you can determine it by using the CF CLI. Run the command:
cf app <APP_NAME>
The resulting output displays information about the application you specified. The output also includes a line starting with “stack:” followed by the name of the stack currently in use.
If cflinuxfs3 is shown here, your application is still using the deprecated stack. If cflinuxfs4 is listed, your application is already using the new stack and no further action is required.
The CF CLI supports the use of third-party plugins. To check which stacks your applications are using, you can use the Stack Auditor CLI Plugin to list applications for each org that you have access to. To see all the applications in your deployment, ensure that you are logged in to the CF CLI as a user who can access all orgs. We do not recommend using the ‘change-stack’ command provided by Stack Auditor plugin. While it can change your stack, it does not provide logs or any information about what went wrong if the operation fails. It also does not work on ARM Macs and uses the deprecated Cloud Foundry API v2. However it works well if you want to find out which of your applications require migration. For more information, see https://docs.cloudfoundry.org/adminguide/stack-auditor.html. Note that this plugin is not supported by SAP.
Alternatively, you can use the jq tool and the CF CLI to query the CF APIs for this information. Note that the jq tool is also not supported by SAP. For more information, see https://stedolan.github.io/jq/.
Using the jq tool, you can run the following script to get an overview of the stack and buildpacks used for all applications within a Cloud Foundry org:
cf curl "/v3/apps?per_page=5000&include=space.organization" | jq '(.included.spaces | INDEX(.guid)) as $spaces | (.included.organizations | INDEX(.guid)) as $orgs | [ .resources[] | {app: .name, org:$orgs[$spaces[.relationships.space.data.guid].relationships.organization.data.guid].name ,space: $spaces[.relationships.space.data.guid].name , lifecycle} ]'
If you want to filter for cflinuxfs3 to display the applications that require manual migration, use:
cf curl "/v3/apps?per_page=5000&include=space.organization" | jq '(.included.spaces | INDEX(.guid)) as $spaces | (.included.organizations | INDEX(.guid)) as $orgs | [ .resources[] | select(.lifecycle.data.stack == "cflinuxfs3") | {app: .name, org:$orgs[$spaces[.relationships.space.data.guid].relationships.organization.data.guid].name ,space: $spaces[.relationships.space.data.guid].name , lifecycle} ]'
2. Determine if Buildpacks Used by Your Applications Support the New Stack
With the CF CLI you can check for the available buildpacks and their stack support by running:
cf buildpacks
Buildpacks supporting cflinuxfs4 will list it in the stack column of the output. The SAP Java Buildpack does not list any stack in the output, but it does support cflinuxfs4.
If you’ve designated a specific buildpack version to be used, it may be necessary to update the buildpack version in the manifest.yml of your application. See https://docs.cloudfoundry.org/devguide/deploy-apps/manifest-attributes.html#buildpacks.
For more information on individual buildpacks, see:
- Staticfile Buildpack
- SAP Java Buildpack
- Java Buildpack
- Ruby Buildpack
- .NET Core Buildpack
- NodeJs Buildpack
- Go Buildpack
- Python Buildpack
- PHP Buildpack
- Nginx Buildpack
- R Buildpack
- Binary Buildpack
3. Deploy an Instance of Your Application Using the New Stack
To check that your application is working properly on the new stack, we recommend that you test it before proceeding with the migration. To do so, push your application a separate time using a new name and a separate route by running:
cf push <APP_NAME> -s cflinuxfs4
If your applications use or contain any precompiled binaries, it is possible that you must recompile them. An example of this are applications depending on binary libraries such as OpenSSL or Python, where cflinuxfs4 relies on newer versions than the ones provided by cflinuxfs3.
If everything worked well, you can continue with your productive apps and change their stack by simply defining it in the cf push command or in the Cloud Foundry manifest. To avoid an unwanted reversion to cflinuxfs3, ensure that your Continuous Deployment automation deploys your applications with cflinuxfs4 as well.
Note that after your application has been deployed using the new stack cflinuxfs4 once, your selection will be saved for future cf push operations. This means that whenever you push the application afterwards, the SAP BTP, Cloud Foundry environment will automatically select the new stack as destination.
Migrate the cflinuxfs4 Stack Using the SAP BTP Cockpit
Important: We identified an issue with the Cockpit switch to cflinuxfs4 and we are working to provide a fix. Please switch to cflinuxfs4 according to the CF CLI instructions for now.
Instead of using the CF CLI, you can switch from cflinuxfs3 to cflinuxfs4 and vice versa in the SAP BTP cockpit.
Prerequisites:
You need to have the Space Developer role assigned to you.
1. Determine Your Current Stack
You can find your current stack listed on the Overview page of your application in the SAP BTP cockpit. Here, the current stack is cflinuxfs3:
2. Determine if Buildpacks Used by Your Applications Support the New Stack
If the Change Stack button appears on the Overview page of your application, this means that the option to switch between cflinuxfs3 and cflinuxfs4 is available.
If you deploy a new application, it should use the new cflinuxfs4 stack by default. Just to make sure that’s the case, you can always check this information on the Overview page of your application.
3. Changing the Stack
To change your current stack, perform the following steps:
- Navigate to the Overview page of your buildpack-based application.
For optimal performance, we recommend that you turn off temporarily the automatic refresh from the upper right-hand corner of the page. You can turn it back on after you change the stack.
- Choose Change Stack.
- Choose a stack from the drop-down list.
The currently selected stack in this example is cflinuxfs3. If you want to migrate to cflinuxfs4, select it from the drop-down list.
If you haven’t turned off the automatic refresh yet, please do so now.
- Please read the instructions carefully. Then, check all the boxes.
- Choose Save. Applying the change may take some time.
Important: If there’s an issue with the switch to cflinuxfs4 and the application stops working, you can revert to cflinuxfs3 by following the same steps. After that, please try to switch to cflinuxfs4 according to the CF CLI instructions.
Hi Beyhan,
thank you for the heads up. But I hope that most deployments use MTAs. As I found in the sample SAP-samples/cf-mta-examples/blob/main/cf-app/mta.yaml#L34 the stack can be configured like this:
Best Regards
Gregor
Hi Gregor,
thank you for the MTA example.
Thank you Gregor. I wouldn't have been able to upgrade the stack without it. For those that do not have a manifest.yml file, you can create it by running the command:
Hi Beyhan,
can you clarify what the force migration exactly means? Will SAP trigger a redeployment at a specific date?
Best Regards
Gregor
Hi Gregor,
I can answer. The force migration will trigger a restage. The application is first stopped, then a new droplet is compiled and then the application is restarted [1].
Regards,
Stephan
[1] https://docs.cloudfoundry.org/devguide/deploy-apps/start-restart-restage.html#restage
Hi all,
please keep in mind that upgrading the version apparently is a pretty memory and storage intensive operation. It might be that you need to increase the quota of the application for the update (you should be able to decrease it afterwards).
At least that was my experience with it. I also had some trouble when upgrading an application via CLI but then I used Gregor's method of specifying it in the mta.yaml and it worked like a charm
Best
Jakob
Hi Jakob,
thank you for your comment. Just to clarify what you mean with "pretty memory and storage intensive operation". Do you refer here to a Blue-Green Deployment or Rolling App Deployments which require additional quota during the deployment?
Regards,
Beyhan
Hi Beyhan,
No I was not using Blue-Green for this deployment as I was just doing it in DEV environment. I just used cf push to update it and it aborted the operation with Error 137 which is out of memory according to my Google searches. When I increased the quota and tried again, it kinda worked. At least it started the app successfully. (Now I am facing an error where apparently OpenSSL is not found. I'll try to fix that tomorrow)
Regards
Jakob
Hi Jakob,
thank you for sharing your experience. I didn't experience noticeable memory usage changes or quota issues with just using cf push on my NodeJS and Java apps but it is good to know. It could be of course application or buildpack specific behaviour. Regarding the OpenSSL issue which buildpack do you use? Do you do anything special with OpenSSL in your application? You can check the OpenSSL version available in the old cflinuxfs3 stack in receipt.cflinuxfs3.x86_64 and for the new cflinuxfs4 in receipt.cflinuxfs4.x86_64.
Regards,
Beyhan
Hi Jakob Ruf ,
Were you able to sort this OpenSSL not found issue? I am facing the same challenge while deploying.
"Error: Connection failed (RTE:[300002] OpenSSL is not available: Unresolved symbol (********-****-****-****-************.hana.prod-eu10.hanacloud.ondemand.com:443))"
Beyhan Veli - any suggestion for this issue? I have changed the buildpack stack in MTA and all services are getting deployed but db-deployer is throwing above error.
Thanks in advance!
Regards
DC
Hi Jakob Ruf, Dhananjay Choubey,
you have to upgrade to the latest HANA client version (at least version 2.16.21). That version supports the upgraded OpenSSL version from the new stack.
Best regards,
Jonathan
Hi Jonathan
How do I get the most recent HANA client version into my cf app? I was following a tutorial at Create a Python Application with Cloud Foundry Python Buildpack | Tutorials for SAP Developers which doesn't cover this.
Thanks
** edit ** for anyone else who ends up finding this, the answer was to change my requirements.txt file so that it has a line:
hdbcli==2.16.21
After doing this, I could then push with -s cflinuxfs4
Thanks Jonathan Bregler .. I updated the npm client version in package.json and we also upgraded the HANA Cloud to latest version. Nothing helped here. Are we doing something wrong here?
Thanks!
Hi Dhananjay Choubey ,
You need to update all instances of @sap/hana-client that are potentially being use in your project, be it directly or indirectly.
If you use the client directly you just have to change the version of @sap/hana-client in your package.json file.
However, you might be using the client indirectly via some other library like @sap/hdbext or @sap/hdi-deploy. In this case you have to update these libraries as well.
A HANA Cloud upgrade is not required. This issue happens entirely on the client side.
Hi Beyhan,
We noticed that "staticfile_buildpack" is not supported in cflinuxfs4 but we do have some applications that use it. How can we deal with this situation before cflinuxfs3 is removed?
OSS message: https://launchpad.support.sap.com/#incident/pointer/002075129500001307412023
Regards,
Jerry Zhang
Hello Jerry,
the "staticfile_buildpack" is not yet supported on cflinuxfs4. I'm optimistic that it will be supported before cflinuxfs3 is removed.
It might be the case, that the "staticfile_buildpack" is not yet available for cflinuxfs4 when cflinuxfs4 becomes the default stack. In this case you need to specify the cflinuxfs3 stack explicitly for new "staticfile_buildpack" applications.
In the unlikely case that "staticfile_buildpack" won't be supported at all on cflinuxfs4, the nginx_buildpack might be an option as it is rather similar to the staticfile_buildpack.
The nginx_buildpack is already available for cflinuxfs4 as online buildpack and we will add it to the offline system buildpacks soon.
Best regards,
Stephan
Any ETA on "nginx_buildpack is already available for cflinuxfs4" availability?
Its just a month till April and we can't deploy nginx_buildpack with cflinuxfs4 without using a custom buildpack.
nginx_buildpack and staticfile_buildpack are now available for cflinuxfs4 as offline system buildpack.
The still missing php_buildpack (for cflinuxfs4, as system buildpack) should arrive ~ 6.4. Until then, you have the option to use the online php-buildpack for cflinuxfs4: https://github.com/cloudfoundry/php-buildpack
I am deploying a docker image using cf push. But I can read in deployment logs:
Is stack also relevant for docker image?
Thanks!
Hi Stephan,
if you follow the CF docs about Deploying an App with Docker you will have at the end an application which shows following:
I wonder why the "cflinuxfs3" is displayed in your case. The stack migration is not relevant for docker apps. Could you check again with the CF docs about Deploying an App with Docker ?
Hi Beyhan, the log is deployed during cloudFoundryDeploy piper step:
But I have the same output than you with
cf app <app_name>
which sounds good.I have tried to force it to
cflinuxfs4
, and received the following error:So it seems that migrating from
cflinuxfs3
(default value, I do not specify it in the manifest) tocflinuxfs4
is not possible. So I just want to be sure that we will not get this error whencflinuxfs4
will become the default value.Hi Stephan,
this error is expected because changing the lifecycle type of an application is not supported an gives the error message you received as described also in the CF API test spec here. Your docker based application will be fine and won't be affected by the stack migration.
Hi Beyhan Veli ,
is changing the CF stack going to cause the application downtime?
Thanks
Arvind Sharma
Hi Arvind,
yes it is. Unless you use blue-green deployment: https://help.sap.com/docs/btp/sap-business-technology-platform/blue-green-deployment-strategy
Regards
Jakob
Hi Arvind,
you could also use Rolling App Deployments feature of CF to switch the stack. The rolling deployment is the native CF approach for zero downtime updates.
Regards,
Beyhan
Hi thanks sharing all the useful information!
I have one question to the cf CLI:
On our btw, the version of the buildback is shown correct.
But via the CLI it still is reporting the fs3 buildback.
Any ideas what im doing wrong?
Regards
Norbert
Hi Norbert,
do you see the cflinuxfs4 stack in the BTP Cockpit and the cflinuxfs3 when you do "cf app <app-name>?
What did you use to switch to the new stack? The Cockpit or the CF CLI?
Regards,
Beyhan
Hi Beyhan,
yes, on the BTP Webpage evrything looks good. Via the CLI ist is still fs3.
As we have multiple apps i tried som changing via the website ans some as an redeploy with the change mta.
Maybe it has something to do with the datacenter? On a other datecenter the everything looks fine (web and CLI)?
regards
Norbert
Hi Norbert,
I think what you noticed is related to the BTP Cockpit UI issue we discovered with the stack switch. All the different datacenter should have the same behaviour. I updated the blog post with a notice about the UI issue.
Regards,
Beyhan
Hi Beyhan,
from our mta.yaml for the sidecar and we have not mentioned the stack and the latest stack has been picked up. Its in CF-EU20.
But we see the below error:
**ERROR** Unsupported stack
Failed to compile droplet: Failed to run all supply scripts: exit status 1
How do we fix this?
Hi Ashwin,
the NodeJS buildpack introduced support for the cflinuxfs4 stack from version >= v1.7.74. Please check the release notes of the NodeJS buildpack. You need to update the currently pinned version "v1.7.54" to a version which has support for the cflinuxfs4 stack.
Regards,
Beyhan
Hello,
I've added new stack to our MTA project file but I'm facing to some error like:
Hi Fabien,
on some apps I also had this issue.
In my case increasing the disk space an then redeploy the MTA fixed it (but I'm not sure if this is really the cause of the issue).
Like described above: https://blogs.sap.com/2023/02/16/deprecation-of-cloud-foundry-stack-cflinuxfs3-and-migration-to-cflinuxfs4/comment-page-1/#comment-661042
Regards
Norbert
Hello Norbert,
How much disk space did you allocate in your project?
For my case I did add 50/100M.
This only happend for me with apps that are near the limit of the disk:quota.
I think the new buildpacks are a little bit larger and therefor they need more space. And if the limit exceeds the quota it breaks.
Thanks, it works 🙂
Hi Fabien,
the error you had:
Hello Beyhan Veli ,
I have very same problem. I have mta deployed app in the subaccount. That app was deployed a long time ago, so it was on cflinuxfs3 stack. And it was working without problems.
Then I changed stack to it (through cockpit) to cflinuxfs4 and I received same error as Fabien up here in thread. I created also question for it - https://answers.sap.com/questions/13855382/error-after-stack-change-to-cflinuxfs4-glibc.html
Then I decided to change stack back to cflinuxfs3, but the same error appeared and app is not able to start. Regardless of linux stack (I tried several times to change it back and forth).
If you would need all logs or some other details, just ask.
Vlado
Hi Vladimír,
we could identify an issue in the Cockpit UI logic which changes the stack to the new one. We are working on a fix. I updated the blog post with a notice about this. Could you try to change the stack via the CLI?
I'm sorry for the inconvenience.
Regards,
Beyhan
Hello Beyhan Veli ,
I tried both way (MTA and cockpit UI) with the same error.
First, based on the reco from Norbert Laenger, I've extend disk space from 1000M to 1200M and then, switch stack to cflinuxfs4 with UI and it was working.
Then, I did the same with MTA and it's working too.