Upcoming breaking change in SAP BTP, Kyma Runtime: Enabling the Istio CNI plugin

I’m part of the Kyma team responsible for Service Mesh and Security topics. My team and I aim to provide a stable environment that enables exposing and securing a workload with ease. The recently introduced Istio Container Network Interface (CNI) plugin, which significantly enhances workload security, will be enabled by default with 2.11 version of Kyma. Let me guide you through the migration process and share the details about this improvement.

Background

So far, workloads being part of Istio Service Mesh were injected with the istio-init container responsible for setting up the networking functionality for the Istio sidecar proxy. Unfortunately, to manage such workloads, their owners needed to have elevated Kubernetes RBAC permissions. The requirement of enabling elevated permissions for every user willing to deploy Pods to Service Mesh caused a security concern that we decided to address. Therefore, we introduced the Istio CNI plugin which does not require the user to have elevated permissions to be able to manage workloads in Service Mesh. Running centrally, it transfers the requirement for elevated permissions from the workload owner to the Service Mesh Namespace. Because the Istio CNI plugin replaces the functionality of the istio-init container, enabling it might cause some network connectivity problems to the custom initContainers relying on network connectivity during the workload initialization phase.

Migration

The Istio CNI plugin will be enabled by default with version 2.11 of SAP BTP, Kyma Runtime scheduled to be released by the end of February 2023. It is already possible to enable the Istio CNI plugin in a development environment and test the upcoming changes there. After successful tests, the migration steps can be replicated in a production environment. Remember that these preparations are mandatory and must be made before Kyma 2.11. To mitigate the risk of workload downtime after the release, take the following steps:
– Verify which workloads may be affected.
– Apply the configuration changes.
– Enable the Istio CNI plugin.
– Verify if workloads are healthy.

To check which of your workloads are prone to face connectivity errors, run:

kubectl get all -o="custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,INIT-CONTAINERS:.spec.initContainers[*].name,CONTAINERS:.spec.containers[*].name,KIND:.kind" -A --field-selector=metadata.namespace!=kyma-system,metadata.namespace!=kube-system,metadata.namespace!=istio-system | awk '{n=split($3,a,/,/); if (length(a)>1) {print}}'

The previous command lists all the workloads outside of the Kyma Namespace and the system Namespace which have the initContainers field defined. Here’s an example of a Pod with the defined initContainers:

apiVersion: v1
kind: Pod
metadata:
  name: example-workload
  namespace: test-namespace
spec:
  containers:
  - name: istio-proxy
    image: eu.gcr.io/kyma-project/external/istio/proxyv2:1.15.3-distroless
    ...
  - image: docker.io/kennethreitz/httpbin
    name: example-workload
    ...
  initContainers:
  - name: example-init-container
    image: eu.gcr.io/kyma-project/external/busybox:1.34.1
    ...
  - name: istio-init
    image: eu.gcr.io/kyma-project/external/istio/proxyv2:1.15.3-distroless
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        add:
        - NET_ADMIN
        - NET_RAW
        drop:
        - ALL
      privileged: false
      readOnlyRootFilesystem: false
      runAsGroup: 0
      runAsNonRoot: false
      runAsUser: 0
  ...

The previous example shows a Pod having the example-workload container and the istio-proxy container. Since this workload is part of Service Mesh it also has the istio-init container with the securityContext which configures the NET_ADMIN and NET_RAW  capabilities as well as elevated values for the runAs* settings. The fact that a workload is listed after executing the command does not mean that it will face errors after the Istio CNI plugin rollout. Each workload should be further analyzed to verify whether its initContainers require network. Only the workloads which do rely on network in the initialization phase need to be configured to mitigate connectivity errors. To eliminate the risk of having networking issues, configure these workloads with one of the following settings:

• Set the UID of the initContainer to 1337  using runAsUser . 1337 is the UID  used by the sidecar proxy. The traffic sent by this UID is not captured by the Istio’s iptables rule. Application container traffic is still captured as usual.
• Set the traffic.sidecar.istio.io/excludeOutboundIPRanges  annotation to disable. It disables redirecting traffic to any CIDRs the initContainers communicate with.
• Set the traffic.sidecar.istio.io/excludeOutboundPorts  annotation to disable. It disables redirecting traffic to the specific outbound ports the initContainers use.

WARNING: Be aware that the excludeOutbound* annotations affect all the containers in a workload, so setting them might introduce issues in those containers that don’t need to be configured.

The recommended configuration should look like this:

apiVersion: apps/v1
kind: Deployment
metadata:
...
  name: example-workload
  namespace: example-namespace
spec:
  ...
  template:
    ...
    spec:
        ...
      initContainers:
      ...
        image: eu.gcr.io/kyma-project/external/busybox:1.34.1
        imagePullPolicy: IfNotPresent
        name: example-init-container
        securityContext:
          privileged: false
          runAsGroup: 1337
          runAsNonRoot: true
          runAsUser: 1337
        ...

When all your workloads have been configured and tested, you are ready to enable the Istio CNI plugin.

To enable the Istio CNI plugin on a cluster, create the following ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kyma-istio-cni
  namespace: kyma-system
data:
  cniEnabled: "true"

Wait up to 15 minutes for the configuration to be applied. Once it is applied, all the workloads in the Service Mesh will be restarted.

Verification

To verify whether the Istio CNI plugin was successfully enabled, see if:

• The istio-initcontainer with the elevatedsecurityContext is gone from your workload.
• The istio-validation initContainer, which checks the correctness of the network setup without the need for elevated permissions, is added to the initContainers field.
• The initContainers do not report any network-related errors.

Here’s an example of a Pod with the Istio CNI plugin correctly enabled:

apiVersion: v1
kind: Pod
metadata:
  name: example-workload
  namespace: test-namespace
spec:
  containers:
  - name: istio-proxy
    image: eu.gcr.io/kyma-project/external/istio/proxyv2:1.15.3-distroless
    ...
  - image: docker.io/kennethreitz/httpbin
    name: example-workload
    ...
  initContainers:
  - name: example-init-container
    image: eu.gcr.io/kyma-project/external/busybox:1.34.1
    securityContext:
      privileged: false
      runAsGroup: 1337
      runAsNonRoot: true
      runAsUser: 1337
    ...
  - name: istio-validation
    image: eu.gcr.io/kyma-project/external/istio/proxyv2:1.15.3-distroless
  ...

Conclusion

The Istio CNI plugin removes the requirement for users deploying workloads to Service Mesh to have elevated privileges. Therefore, it enables administrators to define more fine-grained Kubernetes RBAC permissions. To learn more about the Istio CNI plugin, check the Istio documentation. Read our blog posts to keep up to date with future Kyma releases. If you have any questions about the CNI plugin rollout or other SAP BTP, Kyma Runtime topics, do not hesitate to ask.