Skip to Content
Technical Articles
Author's profile photo Beyhan Veli

Deprecation of Cloud Foundry Stack cflinuxfs3 and Migration to cflinuxfs4

Overview

SAP BTP, Cloud Foundry runtime uses stacks, a prebuilt root file system that in combination with application code and buildpacks is the base to build the application container. The stack in the SAP BTP, Cloud Foundry environment is based on a Linux system and is called cflinuxfs<x>.

Stacks receive regular updates to address security issues and vulnerabilities, which your applications can pick up through new releases and version updates of the SAP BTP, Cloud Foundry Runtime. To get information about version updates and other Cloud Foundry related topics, see What’s New for SAP Business Technology Platform.

Currently in the SAP BTP, Cloud Foundry environment, all applications run on a stack called cflinuxfs3, which is based on Ubuntu Bionic Beaver 18.04. This stack was deprecated in December 2022 and is planned to be removed from the SAP BTP, Cloud Foundry environment in May 2023 after support for Ubuntu Bionic Beaver 18.04 has ended.

The successor stack cflinuxfs4, based on Ubuntu Jammy Jellyfish 22.04, has been available on an opt-in basis since December 2022 and is planned to become the default stack in March 2023.

This change affects anyone who is running buildpack-based applications in the SAP BTP, Cloud Foundry environment.

To help you understand what is going to happen, we’ve created a simplified timeline of the planned changes:

Cloud%20Foundry%20Stack%20Migration%20Timeline

Cloud Foundry Stack Migration Timeline

 

What Happens When cflinuxfs4 Becomes the Default Stack?

Once cflinuxfs4 is designated as the default stack, all new applications will use this stack automatically. Existing applications running in the SAP BTP, Cloud Foundry environment won’t be affected by this change, but it is recommended that you update them to use the new stack by following the guide below, as the old stack will be removed in May 2023 and your app can stop working in case it is not compatible with the cflinuxfs4 stack.

To start using cflinuxfs4 with existing applications, you can push the application and designate the stack manually using:

cf push <APP_NAME> -s cflinuxfs4

If you’re using the blue-green deployment method to push your applications, they are automatically deployed on cflinuxfs4 when it is declared the default stack. This only works if you delete the unused blue or green applications. For more information on blue-green deployment, see https://docs.cloudfoundry.org/devguide/deploy-apps/blue-green.html.

You can continue to use cflinuxfs3 for a transitionary period. This means you can also manually switch back to cflinuxfs3 if you notice problems using the new stack by using the “-s” command when pushing your application:

cf push <APP_NAME> -s cflinuxfs3

cflinuxfs3 will be permanently deleted in May 2023. Afterwards it is no longer possible to manually switch back to cflinuxfs3. For more information on stacks, see https://docs.cloudfoundry.org/devguide/deploy-apps/stacks.html#available-stacks.

Risks and Consequences

If the applications were running, the restart causes a short downtime. However, if your applications are incompatible with the cflinuxfs4 stack, it is possible that they stop working and cannot be restarted.

To maintain functionality and compliance with security standards in the SAP BTP, Cloud Foundry environment, you must migrate your applications to cflinuxfs4. We recommend that you start testing your applications with the new stack as soon as possible to be able to fix potential issues. Only applications that have been migrated to cflinuxfs4 before the force-migration will not suffer from downtimes.

To assist you, we have compiled a short guide that walks you through the required migration steps.

Migrate to the cflinuxfs4 Stack with the CF CLI

Prerequisite:

Make sure you are using a recent version of the Cloud Foundry command line interface (CF CLI). We recommend using version 8. For information about the newest version, see https://github.com/cloudfoundry/cli/releases.

1. Determine Your Current Stack

If you are unsure which stack your applications are currently running on, you can determine it by using the CF CLI. Run the command:

cf app <APP_NAME>

The resulting output displays information about the application you specified. The output also includes a line starting with “stack:” followed by the name of the stack currently in use.

If cflinuxfs3 is shown here, your application is still using the deprecated stack. If cflinuxfs4 is listed, your application is already using the new stack and no further action is required.

The CF CLI supports the use of third-party plugins. To check which stacks your applications are using, you can use the Stack Auditor CLI Plugin to list applications for each org that you have access to. To see all the applications in your deployment, ensure that you are logged in to the CF CLI as a user who can access all orgs. We do not recommend using the ‘change-stack’ command provided by Stack Auditor plugin. While it can change your stack, it does not provide logs or any information about what went wrong if the operation fails. It also does not work on ARM Macs and uses the deprecated Cloud Foundry API v2. However it works well if you want to find out which of your applications require migration. For more information, see https://docs.cloudfoundry.org/adminguide/stack-auditor.html. Note that this plugin is not supported by SAP.

Alternatively, you can use the jq tool and the CF CLI to query the CF APIs for this information. Note that the jq tool is also not supported by SAP. For more information, see https://stedolan.github.io/jq/.

Using the jq tool, you can run the following script to get an overview of the stack and buildpacks used for all applications within a Cloud Foundry org:

cf curl "/v3/apps?per_page=5000&include=space.organization" | jq '(.included.spaces | INDEX(.guid)) as $spaces | (.included.organizations | INDEX(.guid)) as $orgs | [ .resources[] | {app: .name, org:$orgs[$spaces[.relationships.space.data.guid].relationships.organization.data.guid].name ,space: $spaces[.relationships.space.data.guid].name , lifecycle} ]'

If you want to filter for cflinuxfs3 to display the applications that require manual migration, use:

cf curl "/v3/apps?per_page=5000&include=space.organization" | jq '(.included.spaces | INDEX(.guid)) as $spaces | (.included.organizations | INDEX(.guid)) as $orgs | [ .resources[] | select(.lifecycle.data.stack == "cflinuxfs3") | {app: .name, org:$orgs[$spaces[.relationships.space.data.guid].relationships.organization.data.guid].name ,space: $spaces[.relationships.space.data.guid].name , lifecycle} ]' 

 

2. Determine if Buildpacks Used by Your Applications Support the New Stack

With the CF CLI you can check for the available buildpacks and their stack support by running:

cf buildpacks

Buildpacks supporting cflinuxfs4 will list it in the stack column of the output. The SAP Java Buildpack does not list any stack in the output, but it does support cflinuxfs4.

If you’ve designated a specific buildpack version to be used, it may be necessary to update the buildpack version in the manifest.yml of your application. See https://docs.cloudfoundry.org/devguide/deploy-apps/manifest-attributes.html#buildpacks.

For more information on individual buildpacks, see:

 

3. Deploy an Instance of Your Application Using the New Stack

To check that your application is working properly on the new stack, we recommend that you test it before proceeding with the migration. To do so, push your application a separate time using a new name and a separate route by running:

cf push <APP_NAME> -s cflinuxfs4

If your applications use or contain any precompiled binaries, it is possible that you must recompile them. An example of this are applications depending on binary libraries such as OpenSSL or Python, where cflinuxfs4 relies on newer versions than the ones provided by cflinuxfs3.

If everything worked well, you can continue with your productive apps and change their stack by simply defining it in the cf push command or in the Cloud Foundry manifest. To avoid an unwanted reversion to cflinuxfs3, ensure that your Continuous Deployment automation deploys your applications with cflinuxfs4 as well.

Note that after your application has been deployed using the new stack cflinuxfs4 once, your selection will be saved for future cf push operations. This means that whenever you push the application afterwards, the SAP BTP, Cloud Foundry environment will automatically select the new stack as destination.

Migrate the cflinuxfs4 Stack Using the SAP BTP Cockpit

Instead of using the CF CLI, you can switch from cflinuxfs3 to cflinuxfs4 and vice versa in the SAP BTP cockpit.

Prerequisites:

You need to have the Space Developer role assigned to you.

1. Determine Your Current Stack

You can find your current stack listed on the Overview page of your application in the SAP BTP cockpit. Here, the current stack is cflinuxfs3:

2. Determine if Buildpacks Used by Your Applications Support the New Stack

If the Change Stack button appears on the Overview page of your application, this means that the option to switch between cflinuxfs3 and cflinuxfs4 is available.

If you deploy a new application, it should use the new cflinuxfs4 stack by default. Just to make sure that’s the case, you can always check this information on the Overview page of your application.

3. Changing the Stack

To change your current stack, perform the following steps:

  1. Navigate to the Overview page of your buildpack-based application.

For optimal performance, we recommend that you turn off temporarily the automatic refresh from the upper right-hand corner of the page. You can turn it back on after you change the stack.

  1. Choose Change Stack.
  1. Choose a stack from the drop-down list.

 

The currently selected stack in this example is cflinuxfs3. If you want to migrate to cflinuxfs4, select it from the drop-down list.

If you haven’t turned off the automatic refresh yet, please do so now.

  1. Please read the instructions carefully. Then, check all the boxes.
  1. Choose Save. Applying the change may take some time.

Assigned Tags

      74 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Gregor Wolf
      Gregor Wolf

      Hi Beyhan,

      thank you for the heads up. But I hope that most deployments use MTAs. As I found in the sample SAP-samples/cf-mta-examples/blob/main/cf-app/mta.yaml#L34 the stack can be configured like this:

      modules:
        - name: app-module
          parameters:
            buildpack: nodejs_buildpack
            stack: cflinuxfs4

      Best Regards
      Gregor

      Author's profile photo Beyhan Veli
      Beyhan Veli
      Blog Post Author

      Hi Gregor,

      thank you for the MTA example.

      Author's profile photo Francisco Guerrero
      Francisco Guerrero

      Thank you Gregor. I wouldn't have been able to upgrade the stack without it. For those that do not have a manifest.yml file, you can create it by running the command:

      cf create-app-manifest <APP_NAME>
      Author's profile photo Gregor Wolf
      Gregor Wolf

      Hi Beyhan,

      can you clarify what the force migration exactly means? Will SAP trigger a redeployment at a specific date?

      Best Regards
      Gregor

      Author's profile photo Stephan Klevenz
      Stephan Klevenz

      Hi Gregor,

      I can answer. The force migration will trigger a restage. The application is first stopped, then a new droplet is compiled and then the application is restarted [1].

      Regards,
      Stephan

      [1] https://docs.cloudfoundry.org/devguide/deploy-apps/start-restart-restage.html#restage

      Author's profile photo Andreas Jankowiak
      Andreas Jankowiak

      Hello,

       

      cf restage is also triggered when we use the "change stack" in the BTP cockpit, right?

      Best Regards,

      Andreas

      Author's profile photo Jakob Ruf
      Jakob Ruf

      Hi all,

      please keep in mind that upgrading the version apparently is a pretty memory and storage intensive operation. It might be that you need to increase the quota of the application for the update (you should be able to decrease it afterwards).

      At least that was my experience with it. I also had some trouble when upgrading an application via CLI but then I used Gregor's method of specifying it in the mta.yaml and it worked like a charm

      Best

      Jakob

      Author's profile photo Beyhan Veli
      Beyhan Veli
      Blog Post Author

      Hi Jakob,

       

      thank you for your comment. Just to clarify what you mean with "pretty memory and storage intensive operation". Do you refer here to a Blue-Green Deployment or Rolling App Deployments which require additional quota during the deployment?

      Regards,

      Beyhan

      Author's profile photo Jakob Ruf
      Jakob Ruf

      Hi Beyhan,

      No I was not using Blue-Green for this deployment as I was just doing it in DEV environment. I just used cf push to update it and it aborted the operation with Error 137 which is out of memory according to my Google searches. When I increased the quota and tried again, it kinda worked. At least it started the app successfully. (Now I am facing an error where apparently OpenSSL is not found. I'll try to fix that tomorrow)

      Regards

      Jakob

      Author's profile photo Beyhan Veli
      Beyhan Veli
      Blog Post Author

      Hi Jakob,

      thank you for sharing your experience. I didn't experience noticeable memory usage changes or quota issues with just using cf push on my NodeJS and Java apps but it is good to know. It could be of course application or buildpack specific behaviour. Regarding the OpenSSL issue which buildpack do you use? Do you do anything special with OpenSSL in your application? You can check the OpenSSL version available in the old cflinuxfs3 stack  in receipt.cflinuxfs3.x86_64 and for the new cflinuxfs4 in receipt.cflinuxfs4.x86_64.

       

      Regards,

      Beyhan

      Author's profile photo Dhananjay Choubey
      Dhananjay Choubey

      Hi Jakob Ruf ,

       

      Were you able to sort this OpenSSL not found issue? I am facing the same challenge while deploying.

      "Error: Connection failed (RTE:[300002] OpenSSL is not available: Unresolved symbol (********-****-****-****-************.hana.prod-eu10.hanacloud.ondemand.com:443))"

      Beyhan Veli - any suggestion for this issue? I have changed the buildpack stack in MTA and all services are getting deployed but db-deployer is throwing above error.

      Thanks in advance!

       

      Regards

      DC

      Author's profile photo Jonathan Bregler
      Jonathan Bregler

      Hi Jakob Ruf, Dhananjay Choubey,

      you have to use to a somewhat recent HANA client version (at least version 2.12.11). That version supports the upgraded OpenSSL version from the new stack. We recommend that you always update to the latest HANA client version (2.16.21 at the time of writing).

      If you use the HDI deployer you have to upgrade @sap/hdi-deploy to at least version 4.3.3. Again, we recommend that you upgrade to the latest version (4.6.0 at the time of writing) as well.

      Best regards,

      Jonathan

      Author's profile photo Michael W Nicholls
      Michael W Nicholls

      Hi Jonathan

      How do I get the most recent HANA client version into my cf app? I was following a tutorial at Create a Python Application with Cloud Foundry Python Buildpack | Tutorials for SAP Developers which doesn't cover this.

      Thanks

       

      ** edit **  for anyone else who ends up finding this, the answer was to change my requirements.txt file so that it has a line:

      hdbcli==2.16.21

      After doing this, I could then push with -s cflinuxfs4

       

      Author's profile photo Dhananjay Choubey
      Dhananjay Choubey

      Thanks Jonathan Bregler .. I updated the npm client version in package.json and we also upgraded the HANA Cloud to latest version. Nothing helped here. Are we doing something wrong here?

      Thanks!

      Author's profile photo Jonathan Bregler
      Jonathan Bregler

      Hi Dhananjay Choubey ,

      You need to update all instances of @sap/hana-client that are potentially being use in your project, be it directly or indirectly.

      If you use the client directly you just have to change the version of @sap/hana-client in your package.json file.

      However, you might be using the client indirectly via some other library like @sap/hdbext or @sap/hdi-deploy. In this case you have to update these libraries as well.

      A HANA Cloud upgrade is not required. This issue happens entirely on the client side.

      Author's profile photo Joshua Matthews
      Joshua Matthews

      Hi Jonathan Bregler ,

      I am still facing this issue even after updating all libraries to the latest version in my application.

      Error: Connection failed (RTE:[300002] OpenSSL is not available: Unresolved symbol (********-****-****-****-************.hana.prod-us20.hanacloud.ondemand.com:443))

      When I revert back to cflinuxfs3 the application resumes working.

      When I run locally in BAS, I do not encounter an error.

      Are there any other things to try?

      Thanks,

      Joshua

      Author's profile photo Jonathan Bregler
      Jonathan Bregler

      Hi Joshua Matthews ,

      updating the libraries should be enough. If it doesn't work, there is likely still an older version of a library in your application (maybe the node_modules folder or the package-lock.json file weren't updated). I suggest you open a support ticket to have it analyzed.

      BAS doesn't use the CF stack when running locally which is why you don't see the error there.

      Best,

      Jonathan

      Author's profile photo Joshua Matthews
      Joshua Matthews

      The resolution to this was to add/update sap-hdbext-promisfied to the latest version.

      Author's profile photo Marc-Antoine BERTHET
      Marc-Antoine BERTHET

      Hello,

      Thank you for the tips.

       

      We did the following action from our side to make it work :

       

      Open a new terminal on the project

      Update the version of hdbcli in the requirements.txt file :

      hdbcli==2.16.21

       

      Install the sap-hdbext-promisfied (not sure if it was useful) :

      In command line :

      npm install -g sap-hdbext-promisfied

       

      Deploy the update :
      In command line :

      cf push <APP_NAME> -s cflinuxfs4
      Author's profile photo Jerry Zhang
      Jerry Zhang

      Hi Beyhan,

      We noticed that "staticfile_buildpack" is not supported in cflinuxfs4 but we do have some applications that use it. How can we deal with this situation before cflinuxfs3 is removed?

      OSS message: https://launchpad.support.sap.com/#incident/pointer/002075129500001307412023

       

      Regards,

      Jerry Zhang

      Author's profile photo Stephan Merker
      Stephan Merker

      Hello Jerry,

      the "staticfile_buildpack" is not yet supported on cflinuxfs4. I'm optimistic that it will be supported before cflinuxfs3 is removed.

      It might be the case, that the "staticfile_buildpack" is not yet available for cflinuxfs4 when cflinuxfs4 becomes the default stack. In this case you need to specify the cflinuxfs3 stack explicitly for new "staticfile_buildpack" applications.

      In the unlikely case that "staticfile_buildpack" won't be supported at all on cflinuxfs4, the nginx_buildpack might be an option as it is rather similar to the staticfile_buildpack.

      The nginx_buildpack is already available for cflinuxfs4 as online buildpack and we will add it to the offline system buildpacks soon.

      Best regards,
      Stephan

      Author's profile photo Daniel Huser
      Daniel Huser

      Any ETA on "nginx_buildpack is already available for cflinuxfs4" availability?
      Its just a month till April and we can't deploy nginx_buildpack with cflinuxfs4 without using a custom buildpack.

      Author's profile photo Stephan Merker
      Stephan Merker

      nginx_buildpack and staticfile_buildpack are now available for cflinuxfs4 as offline system buildpack.

      The still missing php_buildpack (for cflinuxfs4, as system buildpack) should arrive ~ 6.4. Until then, you have the option to use the online php-buildpack for cflinuxfs4: https://github.com/cloudfoundry/php-buildpack

      Author's profile photo Stephan PRAT
      Stephan PRAT

      I am deploying a docker image using cf push. But I can read in deployment logs:

      stack:               cflinuxfs3

      Is stack also relevant for docker image?

      Thanks!

      Author's profile photo Beyhan Veli
      Beyhan Veli
      Blog Post Author

      Hi Stephan,

      if you follow the CF docs about Deploying an App with Docker you will have at the end an application which shows following:

      cf app <APP_NAME>
      ...
      stack:
      docker image:      <docker-image>:<tag>
      ...

      I wonder why the "cflinuxfs3" is displayed in your case. The stack migration is not relevant for docker apps. Could you check again with the CF docs about Deploying an App with Docker ?

      Author's profile photo Stephan PRAT
      Stephan PRAT

      Hi Beyhan, the log is deployed during cloudFoundryDeploy piper step:

      [2023-03-08T07:45:17.470Z] info  cloudFoundryDeploy -   disk quota:          1G
      [2023-03-08T07:45:17.470Z] info  cloudFoundryDeploy -   health check type:   port
      [2023-03-08T07:45:17.470Z] info  cloudFoundryDeploy -   instances:           2
      [2023-03-08T07:45:17.470Z] info  cloudFoundryDeploy -   memory:              2G
      [2023-03-08T07:45:17.470Z] info  cloudFoundryDeploy -   stack:               cflinuxfs3

      But I have the same output than you with cf app <app_name> which sounds good.

      I have tried to force it to cflinuxfs4, and received the following error:

      [2023-03-08T09:18:14.034Z] info  cloudFoundryDeploy - The app is invalid: Lifecycle type cannot be changed
      [2023-03-08T09:18:14.034Z] info  cloudFoundryDeploy - FAILED

      So it seems that migrating from cflinuxfs3 (default value, I do not specify it in the manifest) to cflinuxfs4  is not possible. So I just want to be sure that we will not get this error when cflinuxfs4  will become the default value.

      Author's profile photo Beyhan Veli
      Beyhan Veli
      Blog Post Author

      Hi Stephan,

      this error is expected because changing the lifecycle type of an application is not supported an gives the error message you received as described also in the CF API test spec here. Your docker based application will be fine and won't be affected by the stack migration.

      Author's profile photo arvind sharma1
      arvind sharma1

      Hi Beyhan Veli ,

       

      is changing the CF stack going to cause the application downtime?

      Thanks

      Arvind Sharma

      Author's profile photo Jakob Ruf
      Jakob Ruf

      Hi Arvind,

      yes it is. Unless you use blue-green deployment: https://help.sap.com/docs/btp/sap-business-technology-platform/blue-green-deployment-strategy

      Regards

      Jakob

      Author's profile photo Beyhan Veli
      Beyhan Veli
      Blog Post Author

      Hi Arvind,

      you could also use Rolling App Deployments feature of CF to switch the stack. The rolling deployment is the native CF approach for zero downtime updates.

       

      Regards,

      Beyhan

      Author's profile photo Norbert Laenger
      Norbert Laenger

      Hi thanks sharing all the useful information!

      I have one question to the cf CLI:

      On our btw, the version of the buildback is shown correct.
      But via the CLI it still is reporting the fs3 buildback.

      Any ideas what im doing wrong?

      Regards
      Norbert

      Author's profile photo Beyhan Veli
      Beyhan Veli
      Blog Post Author

      Hi Norbert,

       

      do you see the cflinuxfs4 stack in the BTP Cockpit and the cflinuxfs3 when you do "cf app <app-name>?

       

      What did you use to switch to the new stack? The Cockpit or the CF CLI?

       

      Regards,

      Beyhan

      Author's profile photo Norbert Laenger
      Norbert Laenger

      Hi Beyhan,

      yes, on the BTP Webpage evrything looks good. Via the CLI ist is still fs3.

      As we have multiple apps i tried som changing via the website ans some as an redeploy with the change mta.
      Maybe it has something to do with the datacenter? On a other datecenter the everything looks fine (web and CLI)?

      regards
      Norbert

      Author's profile photo Beyhan Veli
      Beyhan Veli
      Blog Post Author

      Hi Norbert,

      I think what you noticed is related to the BTP Cockpit UI issue we discovered with the stack switch. All the different datacenter should have the same behaviour. I updated the blog post with a notice about the UI issue.

       

      Regards,

      Beyhan

      Author's profile photo Ashwin S Yadgiri
      Ashwin S Yadgiri

      Hi Beyhan,

      path: mtx-sidecar
          parameters:
            memory: 3G
            disk-quota: 1024M
            instances: 1
            buildpacks:
              - https://github.com/cloudfoundry/nodejs-buildpack.git#v1.7.54

      from our mta.yaml for the sidecar and we have not mentioned the stack and the latest stack has been picked up. Its in CF-EU20.

      But we see the below error:

      **ERROR** Unsupported stack

      Failed to compile droplet: Failed to run all supply scripts: exit status 1

       

      How do we fix this?

      Author's profile photo Beyhan Veli
      Beyhan Veli
      Blog Post Author

      Hi Ashwin,

      the NodeJS buildpack introduced support for the cflinuxfs4 stack from version >= v1.7.74. Please check the release notes of the NodeJS buildpack. You need to update the currently pinned version "v1.7.54" to a version which has support for the cflinuxfs4 stack.

       

      Regards,

      Beyhan

      Author's profile photo Fabien HENIQUE
      Fabien HENIQUE

      Hello,

      I've added new stack to our MTA project file but I'm facing to some error like:

      node: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by node)
      node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by node)
      Any idea on how to fix this?
      Author's profile photo Norbert Laenger
      Norbert Laenger

      Hi Fabien,

      on some apps I also had this issue.
      In my case increasing the disk space an then redeploy the MTA fixed it (but I'm not sure if this is really the cause of the issue).
      Like described above: https://blogs.sap.com/2023/02/16/deprecation-of-cloud-foundry-stack-cflinuxfs3-and-migration-to-cflinuxfs4/comment-page-1/#comment-661042

      Regards
      Norbert

      Author's profile photo Fabien HENIQUE
      Fabien HENIQUE

      Hello Norbert,

      How much disk space did you allocate in your project?

      Author's profile photo Norbert Laenger
      Norbert Laenger

      For my case I did add 50/100M.
      This only happend for me with apps that are near the limit of the disk:quota.
      I think the new buildpacks are a little bit larger and therefor they need more space. And if the limit exceeds the quota it breaks.

      Author's profile photo Fabien HENIQUE
      Fabien HENIQUE

      Thanks, it works 🙂

      Author's profile photo Beyhan Veli
      Beyhan Veli
      Blog Post Author

      Hi Fabien,

       

      the error you had:

      node: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by node)
      node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by node)
      doesn't look to me to be related to disk quota. I would like to find the root cause of the issue. Could you please list the steps you executed before the error? E.g. did you only changed the CF stack in the MTA description and re-deployed? Or did you also try to change the stack via the Cockpit UI?
      Thank you and regards,
      Beyhan
      Author's profile photo Vladimír Balko
      Vladimír Balko

      Hello Beyhan Veli ,

      I have very same problem. I have mta deployed app in the subaccount. That app was deployed a long time ago, so it was on cflinuxfs3 stack. And it was working without problems.

      Then I changed stack to it (through cockpit) to cflinuxfs4 and I received same error as Fabien up here in thread. I created also question for it - https://answers.sap.com/questions/13855382/error-after-stack-change-to-cflinuxfs4-glibc.html

      Then I decided to change stack back to cflinuxfs3, but the same error appeared and app is not able to start. Regardless of linux stack (I tried several times to change it back and forth).

      If you would need all logs or some other details, just ask.

       

      Vlado

      Author's profile photo Beyhan Veli
      Beyhan Veli
      Blog Post Author

      Hi Vladimír,

      we could identify an issue in the Cockpit UI logic which changes the stack to the new one. We are working on a fix. I updated the blog post with a notice about this. Could you try to change the stack via the CLI?

      I'm sorry for the inconvenience.

      Regards,

      Beyhan

      Author's profile photo Fabien HENIQUE
      Fabien HENIQUE

      Hello Beyhan Veli ,

      I tried both way (MTA and cockpit UI) with the same error.

      First, based on the reco from Norbert Laenger, I've extend disk space from 1000M to 1200M and then, switch stack to cflinuxfs4 with UI and it was working.

      Then, I did the same with MTA and it's working too.

      Author's profile photo Beyhan Veli
      Beyhan Veli
      Blog Post Author

      Hello Fabien and Vladimír,

      the Cockpit UI issue is fixed. I couldn't reproduce the MTA issue. In case you can reproduce the issue I would be interested in the output of the following commands:

      cf app <app-name>
      cf app <app-name> --guid
      cf curl "/v3/apps/<app-guid>" #the values of the lifecycle block here

      I suspect that CF allows to run an application droplet with a CF stack for which the droplet wasn't build and opened a community issue for that but as mentioned I couldn't reproduce it.

      Best regards,

      Beyhan

      Author's profile photo Meghana Vyas
      Meghana Vyas

      Thank you for the blog Beyhan. It was very helpful. We encountered error during the migration for applications with nodejs_buildpack. But including below statement in mta.yaml for all modules helped resolve the error.

      stack: cflinuxfs4
      
                    
      Author's profile photo Suchen Oguri
      Suchen Oguri

      Hi Beyhan Veli

      Thanks for the nice explanation. (Thanks Gregor Wolf for the addition about MTA)

      We have a lot of MTA applications and those were deployed using "cf deploy" command instead of "cf push".

      We are trying to identify an option to migrate to cflinuxfs4 centrally with CLI without changing mta.yaml in each MTA project or migrating one by one from BTP cockpit.

      But we don't see any option like "-s" for deploy command. Could you help us to know whether there is any way to do this from CLI with "cf deploy" command as we don't use "cf push"?

      Thanks in advance.

      Regards,

      Suchen

      Author's profile photo Beyhan Veli
      Beyhan Veli
      Blog Post Author

      Hi Leszek Suchenek,

      I'm not aware of an option in the MTA CF CLI plugin to update the stack of an application. If you want to do this in MTA way you will need to define the stack in the mta.yaml.

      If you don't define any stack in the mta.yaml you could use CF CLI / APIs to update the stack and MTA service won't revert this change. What could work is to use the CF APIs to update the stack of the application. Unfortunately, there is no CF CLI command other then "cf push" to do this which I addressed in this issue. Afterwards, the application have to be restaged, which will build an application droplet based on the new stack with a buildpack supporting the new stack and get relaunched with the new droplet. A sample of the commands looks like:

      cf app <my-app> --guid
      cf curl "/v3/apps/<app-guid>" -X PATCH -H "Content-type: application/json" -d '{
          "lifecycle": {
            "type": "buildpack",
            "data": {
              "stack": "cflinuxfs4"
            }
          }
        }'
      cf restage <app-name> --strategy=rolling

      You could also have a look into the stack-auditor plugin which is a CFF community offering and can also update the stack of an application.

      Note: the suggestions and the command sample are a workaround and they need to be validated for your setup. The API request doesn't reconfigure the buildpack used for the application but the expectation is that the "buildpack name" which is currently used supports also cflinxufs4. My recommendation would be to use the mta.yaml for the stack update.

      Author's profile photo Otto Boehrer
      Otto Boehrer

      Hi Beyhan Veli

      our first attempt was to migrate to cflinuxfs4 using the CLI (node app):

      cf push <APP_NAME> -s cflinuxfs4

      That worked perfectly.

      But when tying the same using a manifest-file, the deploy is successful, but the stack remains as is. We added these entries in the manifest file:

        stack: cflinuxfs4
        buildpacks:
        - nodejs_buildpack

      Using CLI command:

      cf push -f my-manifest.yml

      From the text above, we expected that using the manifest comes to the same result, but this is not the case.

      Are we missing something?

      Many thanks!

      Author's profile photo Otto Boehrer
      Otto Boehrer

      The issue is because of my old cf-version. The "recommended" cf version is 8 in the text, but seems to be the minimum mandatory version.

       

      Author's profile photo Beyhan Veli
      Beyhan Veli
      Blog Post Author

      Hi Otto,

      thank you for this information. I found that there was an issue with the stack attribute from CF manifest in the CF CLI version 7 and it was fixed with version "v7.3.0". That means, the minimum version for CF CLI version 7 is "v7.3.0".

       

      Regards,

      Beyhan

      Author's profile photo Beyhan Veli
      Beyhan Veli
      Blog Post Author

      Hi Otto,

      One small update to the CLI version. Our recommended and supported version documented in the BTP help portal is version 8 or newer. The background is that only version 8 uses the new CF API V3 completely. The CF CLI 7 uses for some services related operation the old CF API V2 which are detracted. That is why I would recommend to use the version 8 of the CF CLI.

      Regards,

      Beyhan

       

      Author's profile photo Andreas Jankowiak
      Andreas Jankowiak

      Hello,

       

      does anyone face this error after issuing command:  cf push <apps_name> -s cflinuxfs4 ?

      No start command specified by buildpack or via Procfile.
      App will not start unless a command is provided at runtime.
      Exit status 0
      Uploading droplet, build artifacts cache...
      Uploading build artifacts cache...
      Uploading droplet...
      Uploaded build artifacts cache (130B)
      StagingError - Staging error: Start command not specified
      FAILED

       

      Thank you.

      Author's profile photo Beyhan Veli
      Beyhan Veli
      Blog Post Author

      Hello Andreas,

       

      From the logs you provided I think that the error you get is not related to the CF stack. It looks like CF is not able to configure a start command for your app. I would suggest you to check how CF determines the start command for an app here. Additionally, please check the docs for the command attribute in the app manifest.

      Best regards,

      Beyhan

      Author's profile photo S Patil
      S Patil

      Hi Beyhan,

      I have changed the CF Stack from cflinuxfs3 to cflinuxfs4 through SAP BTP Cockpit for all the three applications we have (app, db-deployer and srv). I have also increased the instance memory and instance disk so as to avoid the memory crash issue. Also, in mta.ymal file, I have added 'stack: cflinuxfs4' under 'parameters' for all the three apps.

      In SAP BTP cockpit, I am able to see the CF stack has been changed to cflinuxfs4. Now, when I am doing manual build of mta.ymal file then its successful; but, deployment of the generated archive file is getting failed. cf dmol log shows below error -

      1. app log -->  a) Error during loading of saas-registry service credentials. saas-registry service is not bound , b) Error during loading of subscription-manager service credentials. subscription-manager service is not bound
      2. db-deployer log --> a) NPM_CONFIG_LOGLEVEL=error , b) Deployment to container <Schema_Name> failed - error: Connection failed
      3. TimeoutError: Acquiring client from pool timed out. Please review your system setup, transaction handling, and pool configuration. (STDERR, APP/PROC/WEB)

      Am I missing any step?

      Thanks,

      Sourabh Patil

      Author's profile photo Beyhan Veli
      Beyhan Veli
      Blog Post Author

      Hi Sourabh,

       

      In general it is fine to change the stack via SAP BTP Cockpit or via the  mta.ymal. One of the two should be enough. I don't see that you missed any step. Regarding the error it is hard for me to get an understanding what happened from the provided logs. Changing the stack in the SAP BTP Cockpit won't touch your service bindings and also, adding only the "stack" parameter to your mta.yaml shouldn't do that. I would suggest you to check out the MTA troubleshooting guid and open a support ticket if it doesn't help.

       

      Best regards,

      Beyhan

      Author's profile photo Tobias Haak
      Tobias Haak

      Hi,

      we have 2 applications running in the same CF space. In the BTP cockpit for one the "Change Stack" button was displayed and I do the migration. Why does the button not show for the second application? Does that mean, that we MUST go via the CLI approach?

      Thanks and regards,

      Tobias

      Author's profile photo Beyhan Veli
      Beyhan Veli
      Blog Post Author

      Hi Tobias,

      the logic of the "Change Stack" button is as follow. If it is shown but disabled that means you are missing the space developer role. If the button isn't shown at all that means the change via it isn't possible. There are situation where the automatic change isn't possible like no buildpack information is available in the lifecycle metadata of the app because CF automatic buildpack detection is used. In this case I would suggest to go via the CLI approach.

       

      Best regards,

      Beyhan

      Author's profile photo Sumit Kumar Kundu
      Sumit Kumar Kundu

      Hi

      Does this affect mobile development kit (MDK) client app deployed in CF too?

      Best regards,

      Sumit

      Author's profile photo Beyhan Veli
      Beyhan Veli
      Blog Post Author

      Hi Sumit,

       

      the MDK client is affected. To my knowledge if you open the Mobile Services admin cockpit with a user with the space developer role the Mobile Services will automatically upgrade to cflinuxfs4. Alternatively you can migrate the stack of the "BindMobileApplicationRoutesToMe" as described in this blog.

      Best regards,

      Beyhan

      Author's profile photo Win Cotiral
      Win Cotiral

      Hi Beyhan,

      I tried to migrate several apps (nodejs_buildpack) using the Change Stack button in the BTP Cockpit. Application Information now shows that the stack is now in cflinuxfs4, but the Package Uploaded indicates FAILED.

      Tried to do a CF restage in the CLI, but it gets an error "Error staging application: BuildpackCompileFailed - App staging failed in the buildpack compile phase".

       

      Best regards,

      Win

      Author's profile photo Beyhan Veli
      Beyhan Veli
      Blog Post Author

      Hi Win,

       

      What happened after the Change Stack action in the BTP Cockpit? Could the apps start successfully? Most probably the applications are not compatible with the new stack and that is why you see the error during restage. In the mean time we have a SAP note about apps incompatible with the new stack which I will recommend you to check. You will need a BTP user to view it but most probably you have one.

       

      Best regards,

      Beyhan

      Author's profile photo Budh Ram
      Budh Ram

      Hi Beyhan,

       

      Is there any exact date decided when SAP will trigger restage of our cf applications?

       

      Thanks,

      Budh Ram

      Author's profile photo Beyhan Veli
      Beyhan Veli
      Blog Post Author

      Hi Budh,

       

      to my knowledge the automated update from the platform side to the new cflinuxfs4 stack will start on 18th of May. The update will take several days that is why there is no exact day. The update will restage an application only if the build process with the new stack is successful. As described in the blog applications which are already running on the new stack won't be touched. Additionally, in the mean time there is also an opt-out possibility from the automated update described in this SAP note. I won't recommend to opt-out because the old stack is deprecated, won't get any updates and will be removed at some point. I also won't recommend to wait for the automated update because of the "Risks and Consequences" described in the blog.

      Best regards,

      Beyhan

      Author's profile photo Beyhan Veli
      Beyhan Veli
      Blog Post Author

      Hi Budh,

      I wanted to provide a correction regarding the date mentioned in my previous comment. The automated update from the platform side to the new cflinuxfs4 stack will actually start on the 17th of May, instead of the 18th. I apologize for any confusion caused by the incorrect date.

       

      Best regards,

      Beyhan

      Author's profile photo Lineker Tomazeli
      Lineker Tomazeli

      I was expecting my stack to be updated automatically by now but nothing has changed. I still see the warning message and the force upgrade has not happened. Any news on this?

      Author's profile photo Beyhan Veli
      Beyhan Veli
      Blog Post Author

      Hi Lineker,

      the automated migration won't touch an application which fails to stage/build with the new cflinuxfs4 stack or will revert it to the old version if it fails to start with the new cflinuxfs4 stack. The migration marks the issue for an app with a label like "failed-restart" or "failed-staging". You should be able to find the labe with "cf labels app <my-app-name>". As mentioned in the blog post there are different reasons why an application could be incompatible with the new stack. This is a case where the app owner have to take care for the stack update and I can just encourage you to do this because the old cflinxufs3 stack reached its end of life.

       

      Best regards,

      Beyhan

      Author's profile photo Lineker Tomazeli
      Lineker Tomazeli

      Hi Beyhan,

      The weird thing is that we tested cflinuxfs4 on the dev environment and everything works fine. So our expectation was that the other environments were going to update automatically.

      Author's profile photo Beyhan Veli
      Beyhan Veli
      Blog Post Author

      Hi Lineker,

       

      have you checked about the mentioned labels? Are they present?

       

      Best regards,

      Beyhan

      Author's profile photo Lineker Tomazeli
      Lineker Tomazeli

      I found a fail-restart label. thanks for your help

      Author's profile photo Sylvain Catudal
      Sylvain Catudal

      Hi Beyhan,

      We changed the stack of each applications that are deployed in the space but still get the warning above the application list. When we navigate to each application details, we see that the stack has been properly migrated: Cloud Foundry Linux-based filesystem (Ubuntu 22.04) (cflinuxfs4)

      We observed the same thing in other environments.  Is this a know issue or there's something we still have to do?

      Thanks

      Author's profile photo Beyhan Veli
      Beyhan Veli
      Blog Post Author

      Hi Sylvain,

       

      this is a generic warning message which informs about the deprecated cflinuxfs3 stack. It doesn't meant that there are apps in the CF environment which are running on that stack. I will provide this feedback to our UI team. Thank you for the feedback.

       

      Best regards,

      Beyhan

      Author's profile photo Sylvain Catudal
      Sylvain Catudal

      Hi Beyhan,

      Thanks for your answer and for bringing it up to the UI team. We are developing a custom solution for a customer who was worried about the warning.

      Thanks again,

      Sylvain