This blog post will guide us through the steps required to configure the SAP Cloud SDK Continuous Delivery Toolkit to scale dynamically on a Kubernetes cluster.
Note: This post is part of a series. For a complete overview please visit the SAP Cloud SDK Overview.
Note: The Kubernetes support in SAP Cloud SDK is currently offered only as an experimental feature.
The current version of the SAP Cloud SDK Pipeline supports autoscaling only if the Jenkins master is also set up on a Kubernetes cluster. To begin with, we need a Kubernetes cluster where we will set up the Jenkins using the Jenkins helm chart. Helm is a package management tool for kubernetes. The documentation to Install Jenkins using helm can be found here. To use the Jenkins image provided by the SAP Cloud SDK, we have to pass s4sdk/jenkins-master as a value for the Master.Image command line argument while deploying Jenkins to Kubernetes.
The successfully completed deployment consists of a Jenkins pod with port 80 and 50000 exposed for HTTP and internal JNLP traffic respectively. The deployment also creates two services each to listen to incoming HTTP traffic on port 80 and the internal JNLP traffic on port 50000. Please note that in this example setup, the SSL/TLS termination happens at the load balancer, hence all the traffic between a load balancer and the Jenkins pod is unencrypted.
The SAP Cloud SDK makes use of the Jenkins Kubernetes plugin. It comes pre-installed with the latest version of the s4sdk/jenkins-master docker image. If we use the helm to install Jenkins then the plugin is automatically configured by the helm chart and we can skip this section. However, if we use any other means to install the Jenkins then, we need to configure the plugin to establish the connectivity to the Kubernetes cluster. To do that navigate to Manage Jenkins menu on the left-hand side of our Jenkins welcome page.
Further navigate to Configure System > Cloud section, configure the Kubernetes cluster details by entering a value to the Kubernetes URL field. Please use the Credentials that have edit access to the namespace which is configured in the Kubernetes Namespace field. Click on Test Connection.
The agents are created as a new pod, a logical node with the collection of containers that are required to execute the pipeline stage. The agent communicates with the master using the JNLP protocol. In this example setup, we are using port 50000 for internal JNLP traffic. Enter the value for the Jenkins tunnel field, which is the service name followed by a port number (jenkins-agent:50000). Please note the tunnel value should not be prefixed with the protocol.
The SAP Cloud SDK Continuous delivery Pipeline needs an environment variable set in the Jenkins to make use of the auto scaling feature. In order to set the environment variable, navigate to Manage Jenkins > Configure System> Global Properties. Add an environment variable ON_K8S and set the value to true.
Now the Jenkins is ready to run our project pipeline as described here. However, every dynamic agent is created as a new pod with a JNLP agent container. By default, the jnlp-agent docker image is used. However, if our Jenkins has a TLS/SSL configuration which terminates at the pod level, then we need to provide a custom JNLP agent image with a valid certificate. Once we publish the custom JNLP image to a secured location, configure the same in our pipeline configuration file pipeline_config.yml.
Note: The Jenkins master server contains a user jenkins with ID 1000. hence, the JNLP agent is expected to have a user with ID 1000 as well to avoid access issues to the files that are shared by both the master and the agent.
That's all the configuration required to benefit from the autoscaling feature of SAP Cloud SDK Pipeline. Push these changes to the repository. Now, the Jenkins agents are dynamically created and resources are allocated on the fly from the Kubernetes cluster for each stage of the pipeline as shown below.
We can test our configuration using the below example Jenkinsfile. Please replace the value for the image if we are using the custom JNLP image.
Please make sure that the URL to the Kubernetes cluster is valid. We can get the URL by executing the kubectl cluster-info command. Make sure that the credential that you are using has the authorization to create and list the pods in a namespace that you have configured.
This issue can arise due to the following two reasons
The service is not created to listen to the JNLP traffic: Please make sure that there is a Kubernetes service which is listening to the JNLP traffic
Misconfigured certificates in Jenkins master and agent: Please make sure that master and the agent uses the same certificate if they have SSL/TLS encryption enabled for communication.
If the Jenkins master and the agent have different user IDs then there will be an issue while accessing files that are created by each other. We might notice errors as shown below.
Please make sure that the Jenkins master and agent have same user IDs. By default, both s4sdk/jenkins-master and s4sdk/jenkins-agent-k8s uses user ID 1000.
The SAP Cloud SDK Continuous Delivery Toolkit enables users to scale the infrastructure on demand. We will benefit from the autoscaling feature only if the infrastructure resources (Kubernetes cluster) are shared among multiple teams (projects) or if we have a Kubernetes cluster that supports autoscaling. However, there could be a noticeable penalty on the performance due to network overhead and pod spin up time if used on a slow infrastructure in comparison to the standalone setup.
Feel free to reach out to us on Stackoverflow via our sap-cloud-sdk tag. We actively monitor
this tag in our core engineering teams.
You can also leave us a comment or a question in the SAP Community using the SAP Cloud SDK tag.
If you would like to report a bug or have an idea for a feature request, you can create a corresponding issue in our GitHub repositories:
Cloud SDK
Cloud SDK Build Pipeline
Note: This post is part of a series. For a complete overview please visit the SAP Cloud SDK Overview.
The goal of this blog post
When we decide to set up our first continuous delivery infrastructure, the first question that arises is, how much resources we need to reserve. However, answering this question is not easy. Having limited resources could throttle concurrent build-pipeline executions. On the other hand, if we plan to reserve enough resources to support concurrent build-pipeline execution, we may end up wasting the resources. There are good chances that, these resources stay idle for the most part of the day.
Autoscaling of the infrastructure solves such a problem. This feature has been introduced in the latest release of the SAP Cloud SDK Pipeline. Instead of reserving the resources proactively, the pipeline creates the Jenkins agents dynamically on a Kubernetes cluster during the execution. Once the agent completes the dedicated task, it is deleted and the resources are freed.
The infrastructure component failure such as a node crash is hard to predict and prevent. If we use the dockerized approach as introduced here, then we need to establish an additional mechanism to manage the failures. For example, we may need an infrastructure health monitoring tool such as Nagios to identify the failure and a fallback script to start services on a backup node.
Thanks to Kubernetes, the monitoring and self-healing come out of the box. Kubernetes performs regular health checks of the infrastructure and the services. If any infrastructure component or the service fails the health check, the kubernetes will create a new component and decommission the old one. If the degraded component is a node then kubernetes will create a new node and also ensures that services that were running on the node are moved to a healthy node. If the Jenkins master pod fails, kubernetes will spin up a new pod and the state is restored by reusing the persistent volume. If a pod fails while executing the pipeline stages, it will be re-created by the Kubernetes without propagating the failure to the pipeline.
In the following sections, we will see how to make use of the autoscaling feature. For a better understanding of this article, please go through the following tutorials first:
Note: The Kubernetes support in SAP Cloud SDK is currently offered only as an experimental feature.
Prerequisite
The current version of the SAP Cloud SDK Pipeline supports autoscaling only if the Jenkins master is also set up on a Kubernetes cluster. To begin with, we need a Kubernetes cluster where we will set up the Jenkins using the Jenkins helm chart. Helm is a package management tool for kubernetes. The documentation to Install Jenkins using helm can be found here. To use the Jenkins image provided by the SAP Cloud SDK, we have to pass s4sdk/jenkins-master as a value for the Master.Image command line argument while deploying Jenkins to Kubernetes.
The successfully completed deployment consists of a Jenkins pod with port 80 and 50000 exposed for HTTP and internal JNLP traffic respectively. The deployment also creates two services each to listen to incoming HTTP traffic on port 80 and the internal JNLP traffic on port 50000. Please note that in this example setup, the SSL/TLS termination happens at the load balancer, hence all the traffic between a load balancer and the Jenkins pod is unencrypted.
Kubernetes Plugin Configuration
The SAP Cloud SDK makes use of the Jenkins Kubernetes plugin. It comes pre-installed with the latest version of the s4sdk/jenkins-master docker image. If we use the helm to install Jenkins then the plugin is automatically configured by the helm chart and we can skip this section. However, if we use any other means to install the Jenkins then, we need to configure the plugin to establish the connectivity to the Kubernetes cluster. To do that navigate to Manage Jenkins menu on the left-hand side of our Jenkins welcome page.
Further navigate to Configure System > Cloud section, configure the Kubernetes cluster details by entering a value to the Kubernetes URL field. Please use the Credentials that have edit access to the namespace which is configured in the Kubernetes Namespace field. Click on Test Connection.
The agents are created as a new pod, a logical node with the collection of containers that are required to execute the pipeline stage. The agent communicates with the master using the JNLP protocol. In this example setup, we are using port 50000 for internal JNLP traffic. Enter the value for the Jenkins tunnel field, which is the service name followed by a port number (jenkins-agent:50000). Please note the tunnel value should not be prefixed with the protocol.
Environment Variable
The SAP Cloud SDK Continuous delivery Pipeline needs an environment variable set in the Jenkins to make use of the auto scaling feature. In order to set the environment variable, navigate to Manage Jenkins > Configure System> Global Properties. Add an environment variable ON_K8S and set the value to true.
Pipeline configuration
Now the Jenkins is ready to run our project pipeline as described here. However, every dynamic agent is created as a new pod with a JNLP agent container. By default, the jnlp-agent docker image is used. However, if our Jenkins has a TLS/SSL configuration which terminates at the pod level, then we need to provide a custom JNLP agent image with a valid certificate. Once we publish the custom JNLP image to a secured location, configure the same in our pipeline configuration file pipeline_config.yml.
Note: The Jenkins master server contains a user jenkins with ID 1000. hence, the JNLP agent is expected to have a user with ID 1000 as well to avoid access issues to the files that are shared by both the master and the agent.
#Project Setup
general:
jenkinsKubernetes:
jnlpAgent: 'custom-jenkins-agent-k8s:latest'That's all the configuration required to benefit from the autoscaling feature of SAP Cloud SDK Pipeline. Push these changes to the repository. Now, the Jenkins agents are dynamically created and resources are allocated on the fly from the Kubernetes cluster for each stage of the pipeline as shown below.
Troubleshooting
Check connectivity to the Kubernetes cluster
We can test our configuration using the below example Jenkinsfile. Please replace the value for the image if we are using the custom JNLP image.
def podName= 'testing'
podTemplate(label: podName, containers: [
containerTemplate(name: 'jnlp', image: 's4sdk/jenkins-agent-k8s:latest'),
containerTemplate(name: 'testcontainer', image: 'alpine:latest', ttyEnabled: true, command: 'cat')
]) {
node(podName) {
stage('Sanity check') {
container('testcontainer') {
echo "I am inside a container"
}
}
}
}Connection to Kubernetes cluster is not established
Please make sure that the URL to the Kubernetes cluster is valid. We can get the URL by executing the kubectl cluster-info command. Make sure that the credential that you are using has the authorization to create and list the pods in a namespace that you have configured.
The agent is created but suspended
This issue can arise due to the following two reasons
The service is not created to listen to the JNLP traffic: Please make sure that there is a Kubernetes service which is listening to the JNLP traffic
Misconfigured certificates in Jenkins master and agent: Please make sure that master and the agent uses the same certificate if they have SSL/TLS encryption enabled for communication.
Cannot create file exception
If the Jenkins master and the agent have different user IDs then there will be an issue while accessing files that are created by each other. We might notice errors as shown below.
sh: 1: cannot create /home/jenkins/workspace/my-project/durable-5bb5cd84/jenkins-log.txt: Permission denied
sh: 1: cannot create /home/jenkins/workspace/my-project/durable-5bb5cd84/jenkins-result.txt.tmp: Permission denied
touch: cannot touch '/home/jenkins/workspace/my-project/durable-5bb5cd84/jenkins-log.txt': Permission denied
Please make sure that the Jenkins master and agent have same user IDs. By default, both s4sdk/jenkins-master and s4sdk/jenkins-agent-k8s uses user ID 1000.
Conclusion
The SAP Cloud SDK Continuous Delivery Toolkit enables users to scale the infrastructure on demand. We will benefit from the autoscaling feature only if the infrastructure resources (Kubernetes cluster) are shared among multiple teams (projects) or if we have a Kubernetes cluster that supports autoscaling. However, there could be a noticeable penalty on the performance due to network overhead and pod spin up time if used on a slow infrastructure in comparison to the standalone setup.
Questions and Feature Requests
Feel free to reach out to us on Stackoverflow via our sap-cloud-sdk tag. We actively monitor
this tag in our core engineering teams.
You can also leave us a comment or a question in the SAP Community using the SAP Cloud SDK tag.
If you would like to report a bug or have an idea for a feature request, you can create a corresponding issue in our GitHub repositories:
Cloud SDK
Cloud SDK Build Pipeline
- SAP Managed Tags:
