Back to the future: Cloud Foundry on Kubernetes
In the Cloud, nothing is as constant as change: Because everything is “on the Internet”, components need to be constantly updated to fix newly found security vulnerabilities. Additionally, Open Source communities are places of vibrant collaboration and their new ideas result in further changes of Open Source technology components. This is of course also true for Cloud Foundry.
At SAP Cloud Platform, we have a history of offering Cloud Foundry as part of our technology portfolio: SAP is one of the founding members of the Open Source Cloud Foundry Foundation created in 2015. At SAPPHIRE 2016, we announced a first beta version of Cloud Foundry as part of what was back then called the SAP HANA Cloud Platform. A year later, again at SAPPHIRE, we announced a generally available version of Cloud Foundry as part of SAP Cloud Platform. Since then, the adoption of Cloud Foundry within SAP and our customer base has increased steadily and so we are meanwhile operating some of the biggest Cloud Foundry deployments worldwide. While we initially mainly consumed Cloud Foundry Open Source upstream, soon several of our colleagues started contributing to Cloud Foundry upstream, many of them parttime, and some even fulltime.
During the same timeframe, a new set of technologies often referred to as container schedulers or container management platforms emerged with Kubernetes establishing itself as de-facto Open Source standard.
In the Open Source community, SAP early on asked the question if it would make sense to combine both Open Source technologies and if so, what a combination of both projects could look like. Together with other major contributor companies we discussed and promoted the idea of combining Cloud Foundry and Kubernetes, for example during Cloud Foundry Summits.
Fast forward two years: Many of our ideas are materializing in the form of new Cloud Foundry Open Source projects taking shape. And so, this blog post describes the trajectory Cloud Foundry is taking towards the future through these projects.
Today’s Cloud Foundry
To level-set the conversation, let’s start by looking at what a typical Cloud Foundry system we run today looks like in terms of its high-level componentry:
First up, there’s BOSH, the lifecycle management tool everybody is using to deploy, update and manage Cloud Foundry. BOSH provides an infrastructure abstraction for the underlying IaaS, allowing us to provision into many public hyperscalers. Next, while we won’t be looking into all the components that make up CF today in detail, it’s important to understand that all of them work in concert to provide the Cloud Foundry user experience and APIs: From being able to use the CF command line tool to providing the Cloud Foundry REST APIs for Cloud Controller and User Account and Authentication server (UAA) to the buildpack logic turning source code into Linux containers (`cf push`), which are then executed by the Diego infrastructure on dedicated VMs which are shared across users of Cloud Foundry.
So, with Diego, today’s Cloud Foundry provides a container runtime for the apps you `cf push` to it.
kubecf: Converting Cloud Foundry to run on Kubernetes
kubecf, a project started and mainly driven by SUSE started with the assumption that Cloud Foundry would continue to evolve based on BOSH and the architecture I described above and that a subset of the Cloud Foundry community wanted to run this evolving stack on Kubernetes. Based on this assumption, it was clear that one needed a “continuous conversion process” consuming new releases of today’s BOSH-managed Cloud Foundry allowing to turn them into artifacts that can be deployed on Kubernetes. The result of this conversion process is called kubecf, a Cloud Foundry distribution that can be deployed on Kubernetes. And the tooling that’s used for this conversion is developed as Project Quarks.
Eirini: Running Cloud Foundry apps on Kubernetes
If you compare the two previous diagrams, you’ll find another difference: Diego got swapped out by Eirini. I already mentioned that with Diego, Cloud Foundry brings an own container scheduler. But Kubernetes itself is a container scheduler as well. So, Eirini is the answer to the question: “Why do we need to run a container scheduler on top of a container scheduler?” with the answer being: “We don’t need to!” So, instead of bringing Diego along to Kubernetes, Eirini allows running Cloud Foundry applications as standard Kubernetes workloads. Via Eirini, Cloud Foundry no longer “runs Cloud Foundry apps” itself but delegates this responsibility to the Kubernetes layer.
As mentioned, SAP has contributors to Cloud Foundry and so it was logical to have contributors to the Eirini project, like other major Cloud Foundry member companies did as well.
cf-for-k8s: A Cloud Native Cloud Foundry distribution
I mentioned that kubecf was built on the assumption that Cloud Foundry would evolve without everybody in the Open Source community embracing Kubernetes. This is no longer the case: Meanwhile, Cloud Foundry is positioned as “The Proven Developer Experience For Kubernetes”. With non-Kubernetes-based Cloud Foundry deployments no longer being the main focus, VMware asked the question what a Kubernetes-native version of Cloud Foundry could look like. The answer is cf-for-k8s, a cloud-native Cloud Foundry distribution. VMware did an exercise of thinking through which components need to remain to still provide the Cloud Foundry user experience (hint: it’s the Cloud Foundry APIs) and for which components there’s a replacement in the broader Kubernetes ecosystem. The result of this exercise looks like this:
The components representing the CF API, Cloud Controller and UAA remain, yet are containerized. Eirini retains its role as Diego replacement as well. What has changed is that the complete Cloud Foundry routing layer (known as gorouter) has been replaced by Istio, the popular service mesh technology from the Kubernetes world. Additionally, logging and monitoring are externalized and components like fluentd are used for log processing and cf-for-k8s exposes the Prometheus monitoring format for its metrics. Lastly, kpack is a component newly built by VMware which retains Cloud Foundry’s buildpack logic, albeit in a Kubernetes-native form by running the buildpack logic on a Kubernetes cluster and creating OCI-compliant container images (the standard container image format Kubernetes expects) which are in turn the input for Eirini. The result is a more componentized distribution of Cloud Foundry, delegating more tasks to existing components from the Cloud Native ecosystem and also allowing consumption of individual components without the need to consume all of Cloud Foundry.
There’s a version of cf-for-k8s which can be run in a local `kind` (Kubernetes in Docker) setup on a laptop. And we have developed an SAP-internal, small-scale deployment we can run on top of project “Gardener” initiated by SAP consuming only limited resources compared to a full-fledged BOSH-deployment of Cloud Foundry.
The role of SAP and summary
While we initially followed the progress of kubecf closely, it became evident that the approach of cf-for-k8s is much more future-oriented and developing faster. Therefore, we decided to solely focus on adopting cf-for-k8s and contributing there as much as we could. Our first contribution steps were to enable running cf-for-k8s on project “Gardener”, but also contributing to Open Source upstream. For instance, by running scalability tests as cf-for-k8s releases were created, and adding fixes as well as features, like the possibility to specify an external database instead of using cf-for-k8s’s built-in one.
Beyond that, we investigate integration topics of cf-for-k8s into SAP Cloud Platform when it comes to topics like for instance lifecycle management, service consumption, authentication and authorization, and commercial aspects.
It is too early to say exactly when the first productive workloads can use cf-for-k8s in an SAP context as well as when – at a later point in time – existing workloads can move from today’s BOSH-managed landscapes to cf-for-k8s without creating big migration efforts for application code, deployment, and tooling.
What is clear, is that Cloud Foundry’s new positioning and joint direction in the open source community will support us in both making sure that new and existing users can continue to work “the Cloud Foundry way” while at the same time evolving the environment in a future-proof direction alongside SAP’s other cloud-based services and solutions – for instance taking advantage of everything Kubernetes has to offer.