Scale your Applications using Custom Metrics Feature of Application Autoscaler

• Is your application unable to cope with large amounts of requests during peak hours?
• Have you seen a decrease in application performance in high-traffic conditions?
• Are you looking to reduce your runtime costs when there is less traffic?
• Do you want to scale your application based on your custom metrics (other than CPU, memory, throughput, reponsetime) ?

Application Autoscaler automatically adds or removes application instances/nodes based on the workload. This can be done by defining an autoscaling policy, containing scaling rules.

In this guide, we will demonstrate how one can use the custom metrics feature (released in app-autoscaler version 3.0.0) using mTLS authentication,

Application Autoscaler Metrics

Let’s quickly discuss the available metrics that Application Autoscaler supports (as of release 9.3.0)

Besides standard metrics, which are mostly available in the container (where the app is running), Application Autoscaler also provides applications to scale out or scale back in based on user-defined metrics types aka. custom metrics. This makes it possible to enhance application performance.

We will use a sample Golang application, deployed on Cloud Foundry which uses the Application Autoscaler custom metrics feature using mTLS authentication. The application can be downloaded from the GitHub repo.

For instance, the application should scale out (add one more instance) if the server is too busy (HTTP 503 Status code) and then scale back in (remove the added instance) if the service is responding with HTTP 200 (normal state).

Custom Metrics Using MTLS

Mutual Transport Layer Security (mTLS) is a certificate-based authentication mechanism, in which both client and server communicate by exchanging x.509 digital certificates. This enables trust and secure communication between them.

Example Scenario

In order to send custom metrics by a cf application to the Application Autoscaler service using mTLS, the following are the high-level steps that need to be performed in the cf application

• Read the app identity certificate and key pair (stored in the app container)
• Create a CA certificate pool and supply certificate to it
• Create an HTTP client and provide the CA certificate pool and certificate
• Send HTTP POST request to autoscaler

The above steps are already implemented in our sample Golang application. Now we have to define our scaling policy which Application Autoscaler uses to perform scaling operations.

Scaling Policy

Our sample Golang application should scale out (add one more instance) if the server is too busy. (HTTP 503 Status code) and then scale in (remove the added instance) if the service responds with HTTP 200.

Prerequisites

Following are the environment and application-specific requirements for this guide.

• BTP account is required. If you don’t have one, try the free trial account
  • Cloud Foundry Runtime entitlement is required
  • Application Autoscaler entitlement is pre-configured
• CF org and space should be available in the Cloud Foundry environment
• CF CLI is installed locally to interact with the Cloud Foundry environment
• Golang application having RESTful API. If you don’t have one, try the sample application from GitHub repo
• Following Autoscaling policy is required

{
        "instance_min_count": 1,
        "instance_max_count": 2,
        "scaling_rules": [
            {
                "metric_type": "tooManyRequestCustomMetrics",
                "breach_duration_secs": 60,
                "threshold": 200,
                "operator": ">",
                "cool_down_secs": 60,
                "adjustment": "+1"
            },
            {
                "metric_type": "tooManyRequestCustomMetrics",
                "breach_duration_secs": 60,
                "threshold": 200,
                "operator": "<",
                "cool_down_secs": 60,
                "adjustment": "-1"
            }
        ]
}

custom-metrics-policy.json

Let’s understand the Application Autoscaler scaling policy

• Scale-out
  • if the metric tooManyRequestCustomMetrics is greater than 200 and it remains greater than 200 for 60 seconds, Autoscaler should perform the scale out (add one more instance/node) of the app.
• Scale-in
  • Similarly, scale in (remove one instance) of an app if the metric tooManyRequestCustomMetrics is less than 200 for 60 seconds. The app instance count remains 1 due to “instance_min_count”: 1.

Walkthrough

Target CF Org and Space

Login to cf and target the cf organization and space, where you want to deploy the application

cf target -o ak_autoscaler_dev -s ak-test-space

Push Golang Application

You need to clone the repository first (if you haven’t yet)

# Clone the Golang Application git repository
git clone https://github.com/SAP-samples/cf-autoscaler-custom-metrics-mtls.git

# Navigate to the directory
cd golang-autoscaler-custom-metrics

# Build application
make build

Let’s push an app to the Cloud Foundry. The following app manifest is used available here

---
applications:
  - name: golang-autoscaler-custom-metrics
    instances: 1
    memory: 128M
    disk_quota: 128M
    stack: cflinuxfs4
    buildpacks:
      - binary_buildpack
    command: ./golang-autoscaler-custom-metrics

cf push --no-start -f deploy/app-manifest.yml -p deploy/build

Noticed, we have not yet started an app. This is because the app needs an Application Autoscaler service instance. This is done in the next step.

Output: cf push an app

Create Autoscaler Service Instance

Now, we want to use the Autoscaler service. So, we create a CF service instance of type autoscaler

cf create-service autoscaler standard ak-test-autoscaler

output: create a service instance

Bind Service Instance with the Golang App

Bind the created ak-test-autoscaler service instance with the Golang app and assign the policy (defined above) as a file

cf bind-service golang-autoscaler-custom-metrics \
                ak-test-autoscaler \
                -c deploy/custom-metrics-policy.json

The console output will suggest to restage golang-autoscaler-custom-metrics. This is because when a service is bound to an existing app, the VCAP_SERVICES environment variable is populated with the information e.g. credentials information of that service.

output: cf bind service

As soon as we bind the service to the Golang app, we can check the bound Autoscaler service.

TIP –  Additionally, Autoscaler plugin can be used to attach the policy. If you have not provided the policy with the cf bind-service command, the following can also be used to attach the policy to the app.

cf attach-autoscaling-policy \
   golang-autoscaler-custom-metrics \
   deploy/custom-metrics-policy.json

Let’s have a look at VCAP_SERVICES environment variable

$ cf env golang-autoscaler-custom-metrics

output: cf env app

In the output, we can see the custom metric mtls_url under the credentials section. Our Golang app uses mtls_url to submit tooManyRequestCustomMetrics custom metric to Autoscaler service.

By using mtls_url, the Golang app uses the certificate-based authentication (mTLS), provided by Autoscaler Custom Metrics feature.

Start Golang App

Start Golang app to consider binding changes

cf start golang-autoscaler-custom-metrics

output: cf start app

Now, our Golang app is running with one instance. We can perform scaling through the Autoscaler service which is waiting to receive custom metrics from the Golang app.

Verify App Scale-Out

We now have to send custom metrics to Autoscaler using the HTTP GET endpoint (provided by the app).

– Grep the route of the app

cf app golang-autoscaler-custom-metrics | grep routes

– Using Postman/Curl, perform the following

$ curl https://golang-autoscaler-custom-metrics.cfapps.sap.hana.ondemand.com/busy/201
{"message":"I am busy with value 201.000000"}%

By running cf app golang-autoscaler-custom-metrics again, we now have two instances of our Golang App (after 60-80 seconds)

output: scale out (2 Instances)

Verify App Scale-In

Now, we want our application to scale in if the requests are below 200 (threshold). Similarly, we also have an HTTP GET endpoint to decrease the instance via curl/postman (whichever you prefer)

$ curl https://golang-autoscaler-custom-metrics.cfapps.sap.hana.ondemand.com/not-busy/190
{"message":"I am not busy with value 190.000000"}%

We now have one instance of our Golang App as required

output: scale in (1 Instance)

Monitor Scaling Events

Sometimes it is important to know how and when scaling events happen. This can be find out using the following three ways

App Autoscaler CLI Plugin

Install the plugin — cf install-plugin -r CF-Community app-autoscaler-plugin
Read more about the CF App Autoscaler CLI Plugin

$ cf autoscaling-history golang-autoscaler-custom-metrics
Retrieving scaling event history for app golang-autoscaler-custom-metrics...
Scaling Type     	Status        	Instance Changes     	Time                          	Action                                                                                	Error
dynamic          	succeeded     	2->1                 	2022-12-30T16:38:08+01:00     	-1 instance(s) because tooManyRequestCustomMetrics <= 200test-unit for 60 seconds
dynamic          	succeeded     	1->2                 	2022-12-30T16:18:08+01:00     	+1 instance(s) because tooManyRequestCustomMetrics >= 200test-unit for 60 seconds

CF App Events

Cloud Foundry provides viewing events of an app. App events can be created, upload, scaled, and so on. For more information on auditing capabilities in Cloud Foundry, have a look at Audit Event on Cloud Foundry documentation.

However, we are only interested to view app scaling events at the moment.

cf events golang-autoscaler-custom-metrics
Getting events for app golang-autoscaler-custom-metrics in org ak_autoscaler_dev / space ak-test-space as ak-user@abc.com...
time                          event                        actor                              description
2022-12-30T16:38:08.00+0100   audit.app.process.scale                                         instances: 1
2022-12-30T16:18:08.00+0100   audit.app.process.scale                                         instances: 2

Autoscaler Application Dashboard

Autoscaler Application Dashboard is a CF app available to SAP customers that comes out of the box when you use the Application Autoscaler service.

Conclusion

• We created a sample Golang app that sends metrics to the Autoscaler service.
• We learned how to use mTLS-based authentication to send custom metrics.
• The application was scaled out and scaled in based on the metrics sent by the Golang app.
• Additionally, we also learned how to monitor scaling events using different methods and tools.

References

• Learn more about Application Autoscaler Custom Metrics feature on sap.com
• Manage policies, retrieve metrics and scaling event history using app-autoscaler-cli-plugin
• How does Autoscaler decide to shutdown an instance? 
• Get to know about Autoscaler pricing