Technical Articles
Proof of Concept: Deploying ABAP in Kubernetes
Richard Treu, Henning Sackewitz
This blog post describes our experiences and findings while doing a proof of concept (PoC) in the SAP LinuxLab where we containerized the SAP ABAP application server components and deployed them on various Kubernetes environments. It will also point out potential benefits and challenges.
Please note that this document is neither a complete solution, nor does it provide any current product or development status. The current support status of ABAP application servers running in a container(-orchestration) is documented in SAP note 1122387.
Feel free to comment and share this blog post.
Scope
Every ABAP system consists of three tiers: the database containing data and programs, the application server and the clients. For this PoC the scope was on the ABAP application server.
The SAP NetWeaver Application Server ABAP can be broken up into multiple components and thus are subjects for containerization. The first natural choice for containerizing would be the Application Server
instances (AS) because they are the most stateless, cattle-like part of the stack, and they can be scaled relatively easily. However, we also opted for deploying the mandatory components of the ABAP Central Services
, namely the Message Server
and the Enqueue Server
. Finally we added the optional SAP Web Dispatcher
and SAProuter
to the setup.
The underlying SAP HANA database was out of scope for our PoC – it was regarded as a given external resource that can be connected to via configurable secure store credentials.
Our effort was to put all the above components into separate container images, map them to appropriate Kubernetes objects and tie them together in a way that we can use Kubernetes features in the best way.
Goal
Our goal was to create a generic ABAP Kubernetes deployment that can integrate into any Kubernetes environment, regardless if it is an on-premise, self-managed Kubernetes-based product (e.g. CaasP, OpenShift) or a Kubernetes as a service offering in the public cloud (e.g. GKE).
Setup
Docker Images and Kubernetes Deployment Files
In Kubernetes, applications are distributed via pre-built container images along with Kubernetes YAML deployment files.
Our goal was to build generic ABAP images that can be customized with environment-specific input parameters which are configurable via Kubernetes YAML files. At deployment time they are injected into the Kubernetes environment. For example, the HANA database connection parameters will be injected as Kubernetes secrets.
Some attributes are static, immutable values and not configurable in this PoC:
- SAP SID
- SAP instance number
- SAP admin user
Also, we selected the most current, backward-compatible SAP kernel that will work with most NetWeaver and S/4HANA releases.
Deployment in Kubernetes
Application Server (AS)
The actual workload in an ABAP system is performed on the Application Server
in a server-side session. This is where most memory and processing power will be consumed aside from the database, so this is the most important entity to be scaled with Kubernetes according to workload demand.
It is very easy to scale up application server instances (Pods
) as workload grows, but the scale-down can lead to broken user sessions on the system if Pods
are just arbitrarily destroyed in a cattle-like manner.
We placed the application server in a Deployment
with one initial replica. A Deployment
can be scaled-down in a user-controlled order, as opposed to a StatefulSet
where only a reverse-ordered Pod
scale-down is possible, regardless of actual user session load.
We solved the hard shutdown issue by implementing a Horizontal Pod Autoscaler
logic: priority Annotations
are assigned to the Pods
according to their current session load. Whenever a scale-down is being executed, the server with the lowest priority will be issued a soft-shutdown, and sessions will be slowly drained from the Pod
.
As the application work processes are producing several log files, a sidecar container is used to pull the logs and forward them to a log target for each Application Server
Pod
. This way log files are persisted, e.g. for root cause analysis after work process failures and subsequent container restarts.
Message Server and Enqueue Server
The Message Server
is a singleton instance per design, as well as the Enqueue Server
. For greater flexibility we created separate container images for each, but placed them inside one Pod
called ASCS
(ABAP SAP Central Services
).
Since it is necessary for an Application Server
to reach the Message Server
via a static DNS name, we placed the ASCS
Pod
in a StatefulSet
which makes it resolvable.
Since the Message Server
is basically stateless, a container restart is not critical. The Enqueue Server
keeps the lock table so it is not completely stateless. To implement high availability for the Enqueue Server
it is recommended to start a secondary enqueue server that keeps a copy of the lock table. This is known as Enqueue Replication
and could be achieved by creating another singleton Pod
. However, this was out of scope for this PoC so far.
SAProuter and Web Dispatcher
For accessing the system via SAP GUI
, the SAProuter
is able to connect a client to the correct application server. In contrast to the Kubernetes load balancers, the SAProuter
is aware of the proprietary SAP DIAG protocol and forwards connections to the corresponding sessions. The SAProuter
is stateless and can be scaled easily if necessary. It can be deployed as a Pod
, DaemonSet
or Deployment
.
The last component is the Web Dispatcher
which is a load balancer enhanced with proprietary security features and endpoint control. It is stateless and can be scaled up easily if needed. Since we needed only one Web Dispatcher
instance in our PoC, we bundled it together with the Message Server
and the Enqueue Server
into the same Pod
.
Note: It is possible to skip the Web Dispatcher
and use the Kubernetes load balancer to connect to the ICM
(Internet Communication Manager
) processes of the application server containers directly – this is however critical from a security perspective and would constitute a non-standard SAP setup.
Communication and Client Connectivity
After all relevant SAP components were organized in Kubernetes Pods
, we had to make sure that they can properly communicate with each other, as well as with external clients.
Services
Communication between Pods
in a Kubernetes cluster is done via Services
. Since Kubernetes does automatic port mapping on a Node
where multiple pods expose identical ports, this setup allows SAP application server scale-up on a single Node
without port conflicts.
Both the Application Server
Deployment
and the ASCS
StatefulSet
were encapsulated in Kubernetes Services.
Load Balancer
Connections from external clients (SAP GUI, web browser) to the Services is done via an external load balancer. The load balancer type depends on the underlying infrastructure that Kubernetes is running on. For this PoC we used OpenStack with a HAProxy
load balancer, as well as a bare-metal infrastructure. Deploying the load balancer requires API calls into the IaaS layer, so the IaaS-specific Kubernetes Cloud Provider Interface
(CPI) has to be configured. For simplicity, we used MetalLB as load balancer in the end. We successfully tested HAProxy
and a hardware load balancer, as well.
The external load balancer IP resp. its DNS-resolvable host name is the single entry point for all client communication.
The load balancer does not actually do as its name implies. In this setup it is just used as an external communication entry point. In fact, the load is distributed by the Web Dispatcher
and Message Server
using SAP Logon Groups.
Namespaces
Finally, we organized all SAP Kubernetes objects in a dedicated Kubernetes Namespace
‘sap’ for logical separation from other cluster artifacts.
Furthermore, multiple SAP instances could be deployed on a single cluster by assigning them to separate Namespaces, e.g. ‘sapqa’, ‘sapdev’, ‘sapprod’.
PoC Architecture
So here is a picture of how it all comes together:
Conclusions
In principal it is possible to run ABAP in a Kubernetes environment. It allows rapid and flexible deployments especially for test, development and training systems. Due to the comprehensive architecture of the ABAP application server components, some challenges and overlaps with Kubernetes functionalities exist and must be addressed accordingly (e.g. load balancing, name resolution, lifecycle).
Benefits
No installation procedure required
Thanks to the pre-built container images there is no need to install every new ABAP instance as traditionally done on bare metal hardware or virtual machines. We just provide a collection of Kubernetes deployment files and some container images which are dynamically deployed within the Kubernetes cluster.
(Re-)Deploy ABAP instances in a matter of seconds
Once a container image is downloaded and cached, Kubernetes will bootstrap complete ABAP systems in a very short amount of time. All ABAP containers will be orchestrated across the available Kubernetes Worker Nodes
automatically whenever there is a service disruption (e.g. hardware outage).
Scale a small system large with just one click
Separating the scalable Application Server
from the ASCS by placing them in dedicated containers allows for spinning up multiple SAP dialog instances with one command or one click. Because of the encapsulated design of dialog instances and the usage of virtual service endpoints in Kubernetes, scaling up ABAP systems is pretty easy.
Auto-Scaling of Application Servers
Kubernetes standard features include automatic scaling of Pods
based on CPU utilization or memory pressure. These auto-scaling functions can be leveraged to elastically scale an ABAP system when detecting very high or low load. Shared hardware resources in the customers data center can then be utilized more efficiently, especially for non-productive systems without any live intervention of an administrator.
Deployment of multiple, adjacent landscapes
Another benefit is the simple and fast deployment of multiple ABAP instances in the same environment. It is possible to spin up an ABAP instance either in a single Kubernetes cluster, or to share a Kubernetes cluster with multiple ABAP instances. All ABAP instances will be available via load balancer addresses provided by the underlying infrastructure (on-premise/self-managed or public cloud). Kubernetes also takes care of port mapping and avoids conflicts between SAP instances with identical ports on the same Node
, by assigning unique intermediate ports.
Challenges
Auto-Scaling vs. Session-Stickiness
The ABAP architecture keeps user session contexts on one specific dialog instance server during the whole user session until either the user logs off or the session reaches a timeout. A scale-down of dialog instance servers can lead to terminated user sessions.
Furthermore, batch processes – which also live on dialog instance servers – must not be terminated. In our PoC we solved this through a prioritization mechanism to determine which container can be terminated.
Load-Balancing Mechanisms
One benefit of Kubernetes is the built-in load balancing between Worker Nodes
. However, ABAP provides its own load-balancing and enqueue mechanisms according to the used access method (SAP GUI, Web GUI, RFC, …). Thus, there is a function overlap, and Kubernetes load-balancing can only be used in a limited manner.
Raised Complexity for System Connectivity
Containerization and the underlying infrastructure platform add multiple network layers, so accessing the SAP system from a client (SAP GUI, browser) is more complex than accessing bare-metal systems. On the other hand, Kubernetes tools give the ability to permanently check the system availability and network perfomance to identify issues.
Database with compatible SAP NetWeaver or S/4HANA Content required
The database holds ABAP programs, all the business logic and all customer data. To connect a containerized ABAP system with a specific kernel, a compatible SAP HANA database with the correct initial database load is required.
Application specific requirements
We assume that the SAP application server and its ABAP applications may have further requirements, e.g. web service endpoints, remote system connections or mobile application connectivity. There are also implicit assumptions about the underlying infrastructure, e.g. hardware IDs for the SAP license key; Linux kernel parameter values, etc.
Great blog!
Was a working Proof-of-Concept built? If yes, any documents detailing some of the steps taken?
Conceptually I see how this could work, and the blog does a great job of explaining that. I'm wondering if there is anything on the practical side of the steps that were taken.
This would be something really good for operations! Is not the first time I read about dockerizing ABAP AS, but this time contains about all major components in productive scenario and proposing how to orchestrate them.
PS.: Could you fix the image? it is pointing to SAP intranet address.
Thanks for your comment. The image is fixed now.
Dear Richard Treu,
thanks for the blog! It's great to see efforts by SAP (even if "only as a PoC in SAP Labs)!
As a developer outside of SAP I've got no chance of containerizing except packing the AS ABAP as a whole into a 15GB image. Not really a great experience.
I'd wish for the ABAP guys to use your PoC as an alternative DEV edition (looking at you Jens Weiler & Thomas Fiedler)!
A second thing coming to my mind:
I'm currently becoming a huge fanboy of OpenFaaS and the PLONK-stack. OpenFaaS utilizes Prometheus for scaling (which is more flexible compared to a normal HPA) and includes Linkerd which supports traffic shifting as of 2.4. I'm not really in-the-know about that, but it might be a starting point for you to investigate further
hi Oliver Jaegle,
openfaas is serverless (function as a service) which is beyond what netweaver or ERP system work as POD under docker using kuberntes , offcourse openfaas sits on top of kubenetes and underline docker which you can seamless use certain type of functions like "RFC connections, calling ABAP Reports, take Database backup and restore with functions " but overall you will not be able to work like 100% traditional ABAP environment using OPENFAAS.
Serverless have difference use cases.
The title is somewhat misleading because ABAP code still cannot be deployed, it is living inside the database and has to be transported there. The fact that ABAP code is not file based and that there is no build - deploy - circle for ABAP development is a large impediment for becoming more flexible.
Maybe code could be promoted via abapGit?
Hi Rolf, Sèrgio,
yes, the PoC described in the article does not touch the ABAP Code. There is another project called "Decentralized Development" for that, see https://wiki.wdf.sap.corp/wiki/pages/viewpage.action?pageId=2140756912
The infrastructure is ready, an isolated ABAP development system can be technically deployed within minutes (also on Kubernetes, but developers usually don't care). Organizational onboarding takes of cause some time as git is quite different to the usual ABAP change management procedures.
You (your team) can be a part of a pilot for git-based ABAP development for S4HANA.
Just drop me a mail If you are interested.
Bets regards, Alexander
Thanks Alex, I gave the info to a former colleague who is working with ABAP. But I am am not sure if this will have any future. The centralized and monolithic development is buried too deeply inside the ABAP platform. Look at proprietary dev tools and frameworks like BOPF that besides code produce unreadable entries in ABAP tables. Or the lack of layers in so called modern ABAP development that make it still necessary to transport database structure and code together.
Like ABAP in the cloud it may work for a restricted ABAP derivate, but developers who want to build software for the future are better off with any other language like Java or Python that is for years at a level that ABAP is struggling to reach in PoCs and pilot projects.
Well, it up to everybody, to use ABAP or not. My personal opinion: yes, ABAP has future, and I am not talking about a range on 1 year.
The point is: if somebody wants to use ABAP today and enjoy comfort of GIT at the same time, welcome!
Interesting. I'm playing with the same idea myself.
Sadly that page is not accessible, could you please give me more details?
Marcello
Alexander Kosolapov
I would like to be part of the ABAPGit pilot, is there a email address I can use?
Regards,
Ricardo
Hello Alexander Kosolapov,
since we have now the SAP ABAP Platform 1909 Developer Edition being shipped as Docker image, i'm wondering what else happened since you published this great blogpost about your PoC.
Can we expect container support for at least small SAP components like SAP Web Dispatcher, SAPRouter, SAP Cloud Connector, SAP BC, etc. any soon?
Best regards,
Joe