Skip to Content
Technical Articles
Author's profile photo Bernd Krannich

Back to the future: Cloud Foundry on Kubernetes

In the Cloud, nothing is as constant as change: Because everything is “on the Internet”, components need to be constantly updated to fix newly found security vulnerabilities. Additionally, Open Source communities are places of vibrant collaboration and their new ideas result in further changes of Open Source technology components. This is of course also true for Cloud Foundry.

Introduction

At SAP Cloud Platform, we have a history of offering Cloud Foundry as part of our technology portfolio: SAP is one of the founding members of the Open Source Cloud Foundry Foundation  created in 2015. At SAPPHIRE 2016, we announced a first beta version of Cloud Foundry as part of what was back then called the SAP HANA Cloud Platform. A year later, again at SAPPHIRE, we announced a generally available version of Cloud Foundry as part of SAP Cloud Platform. Since then, the adoption of Cloud Foundry within SAP and our customer base has increased steadily and so we are meanwhile operating some of the biggest Cloud Foundry deployments worldwide. While we initially mainly consumed Cloud Foundry Open Source upstream, soon several of our colleagues started contributing to Cloud Foundry upstream, many of them parttime, and some even fulltime.

During the same timeframe, a new set of technologies often referred to as container schedulers or container management platforms emerged with Kubernetes establishing itself as de-facto Open Source standard.

In the Open Source community, SAP early on asked the question if it would make sense to combine both Open Source technologies and if so, what a combination of both projects could look like. Together with other major contributor companies we discussed and promoted the idea of combining Cloud Foundry and Kubernetes, for example during Cloud Foundry Summits.

Fast forward two years: Many of our ideas are materializing in the form of new Cloud Foundry Open Source projects taking shape. And so, this blog post describes the trajectory Cloud Foundry is taking towards the future through these projects.

Today’s Cloud Foundry

To level-set the conversation, let’s start by looking at what a typical Cloud Foundry system we run today looks like in terms of its high-level componentry:

Cloud%20Foundry%20today

First up, there’s BOSH, the lifecycle management tool everybody is using to deploy, update and manage Cloud Foundry. BOSH provides an infrastructure abstraction for the underlying IaaS, allowing us to provision into many public hyperscalers. Next, while we won’t be looking into all the components that make up CF today in detail, it’s important to understand that all of them work in concert to provide the Cloud Foundry user experience and APIs: From being able to use the CF command line tool to providing the Cloud Foundry REST APIs for Cloud Controller and User Account and Authentication server (UAA) to the buildpack logic turning source code into Linux containers (`cf push`), which are then executed by the Diego infrastructure on dedicated VMs which are shared across users of Cloud Foundry.

So, with Diego, today’s Cloud Foundry provides a container runtime for the apps you `cf push` to it.

kubecf: Converting Cloud Foundry to run on Kubernetes

kubecf, a project started and mainly driven by SUSE started with the assumption that Cloud Foundry would continue to evolve based on BOSH and the architecture I described above and that a subset of the Cloud Foundry community wanted to run this evolving stack on Kubernetes. Based on this assumption, it was clear that one needed a “continuous conversion process” consuming new releases of today’s BOSH-managed Cloud Foundry allowing to turn them into artifacts that can be deployed on Kubernetes. The result of this conversion process is called kubecf, a Cloud Foundry distribution that can be deployed on Kubernetes. And the tooling that’s used for this conversion is developed as Project Quarks.

Eirini: Running Cloud Foundry apps on Kubernetes

If you compare the two previous diagrams, you’ll find another difference: Diego got swapped out by Eirini. I already mentioned that with Diego, Cloud Foundry brings an own container scheduler. But Kubernetes itself is a container scheduler as well. So, Eirini is the answer to the question: “Why do we need to run a container scheduler on top of a container scheduler?” with the answer being: “We don’t need to!” So, instead of bringing Diego along to Kubernetes, Eirini allows running Cloud Foundry applications as standard Kubernetes workloads. Via Eirini, Cloud Foundry no longer “runs Cloud Foundry apps” itself but delegates this responsibility to the Kubernetes layer.

As mentioned, SAP has contributors to Cloud Foundry and so it was logical to have contributors to the Eirini project, like other major Cloud Foundry member companies did as well.

cf-for-k8s: A Cloud Native Cloud Foundry distribution

I mentioned that kubecf was built on the assumption that Cloud Foundry would evolve without everybody in the Open Source community embracing Kubernetes. This is no longer the case: Meanwhile, Cloud Foundry is positioned as “The Proven Developer Experience For Kubernetes”. With non-Kubernetes-based Cloud Foundry deployments no longer being the main focus, VMware asked the question what a Kubernetes-native version of Cloud Foundry could look like. The answer is cf-for-k8s, a cloud-native Cloud Foundry distribution. VMware did an exercise of thinking through which components need to remain to still provide the Cloud Foundry user experience (hint: it’s the Cloud Foundry APIs) and for which components there’s a replacement in the broader Kubernetes ecosystem. The result of this exercise looks like this:

cf-for-k8s

The components representing the CF API, Cloud Controller and UAA remain, yet are containerized. Eirini retains its role as Diego replacement as well. What has changed is that the complete Cloud Foundry routing layer (known as gorouter) has been replaced by Istio, the popular service mesh technology from the Kubernetes world. Additionally, logging and monitoring are externalized and components like fluentd are used for log processing and cf-for-k8s exposes the Prometheus monitoring format for its metrics. Lastly, kpack is a component newly built by VMware which retains Cloud Foundry’s buildpack logic, albeit in a Kubernetes-native form by running the buildpack logic on a Kubernetes cluster and creating OCI-compliant container images (the standard container image format Kubernetes expects) which are in turn the input for Eirini. The result is a more componentized distribution of Cloud Foundry, delegating more tasks to existing components from the Cloud Native ecosystem and also allowing consumption of individual components without the need to consume all of Cloud Foundry.

There’s a version of cf-for-k8s which can be run in a local `kind` (Kubernetes in Docker) setup on a laptop. And we have developed an SAP-internal, small-scale deployment we can run on top of project “Gardener” initiated by SAP consuming only limited resources compared to a full-fledged BOSH-deployment of Cloud Foundry.

The role of SAP and summary

While we initially followed the progress of kubecf closely, it became evident that the approach of cf-for-k8s is much more future-oriented and developing faster. Therefore, we decided to solely focus on adopting cf-for-k8s and contributing there as much as we could. Our first contribution steps were to enable running cf-for-k8s on project “Gardener”, but also contributing to Open Source upstream. For instance, by running scalability tests as cf-for-k8s releases were created, and adding fixes as well as features, like the possibility to specify an external database instead of using cf-for-k8s’s built-in one.

Beyond that, we investigate integration topics of cf-for-k8s into SAP Cloud Platform when it comes to topics like for instance lifecycle management, service consumption, authentication and authorization, and commercial aspects.

It is too early to say exactly when the first productive workloads can use cf-for-k8s in an SAP context as well as when – at a later point in time – existing workloads can move from today’s BOSH-managed landscapes to cf-for-k8s without creating big migration efforts for application code, deployment, and tooling.

What is clear, is that Cloud Foundry’s new positioning and joint direction in the open source community will support us in both making sure that new and existing users can continue to work “the Cloud Foundry way” while at the same time evolving the environment in a future-proof direction alongside SAP’s other cloud-based services and solutions – for instance taking advantage of everything Kubernetes has to offer.

Assigned tags

      3 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Denys van Kempen
      Denys van Kempen

      Excellent blog post, Bernd!

      Clear overview of current trends and SAP’s choice of direction.

      ===

      For the reader / those interested in the story about how Cloud Foundry was introduced with the acquisition of SuccessFactors back in 2011/12 and how the SAP Cloud PaaS Platform evolved, see

      About how Hybris introduced Kubernetes and how this relates to Cloud Foundry

       

      Source: https://github.com/EngineerBetter/k8s-is-not-a-paas

      Author's profile photo Johan Groesbeek
      Johan Groesbeek

      Hello ! Thank you very much for the excellent explanation !

      Question  How about Kyma  in this concept ?    :  https://kyma-project.io/blog/

      Author's profile photo Bernd Krannich
      Bernd Krannich
      Blog Post Author

      Hello Johan,

      This is an excellent question. First of all, it’s important to understand that from a developer’s perspective, the two runtime environments have a different focus:

      • Cloud Foundry is a runtime that lets developers focus on their application development. It’s a runtime for stateless workloads abstracting away the “underlying infrastructure details” and allowing developers to focusing on their application coding leaving the “code to container” workflow to the Cloud Foundry runtime. Today, these underlying details are the management of IaaS resources like VMs, networks, etc. Tomorrow, with cf-for-k8s these underlying details are the management of Kubernetes clusters and Kubernetes resources. The initial goal of transitioning towards CF on Kubernetes is therefore to retain the existing Cloud Foundry abstraction, thereby essentially not changing today’s developer workflow.
      • Kyma is the SAP runtime environment that gives more flexibility but requires more control in return. It is a runtime on top of Kubernetes, but different from CF in that Kyma is not abstracting away Kubernetes as a runtime. This means that you can deploy a broader set of workloads compared to Cloud Foundry (including stateful ones) but in exchange will need to work with the underlying low-level questions like how you build your container image, what do you put into it and how do you keep all assets up-to-date, how do translate your application into Kubernetes resources, etc.

      In the end, it is not about which one is better, but it is about which runtime best fits your needs and skills.

      This is the consumer perspective. Additionally, from our perspective of providing and operating both runtimes, there’s a broad set of topics that both runtimes have in common specifically because they are running on the same infrastructure. Our teams are closely collaborating on these commonalities wherever they can. The goal is to make sure that developers don’t have to take a strict either/or decision for one runtime and then must stick to it. Instead, it should be possible to realize scenarios that have assets powered by both runtimes, with these assets being able to interoperate seamlessly. Stay tuned for more information on the topic.

      Regards,
      Bernd