Receive alerts from Kibana enhanced with Open Distro using SAP Cloud Platform Alert Notification

Have you established an application monitoring based on open source Kibana distribution? And do you already use the alerting capabilities of SAP Cloud Platform Alert Notification service? If so, there’s an interesting scenario for you – integrate them both, reuse your preferred communication channel and be always alarmed for any potential issue in your solution.

To complete the story below, you would only need to enhance your Kibana instance with OpenDistro if you haven’t done it so far. Throughout the blog post, we’ll set up the alerting feature to notify us anytime an error threshold is reached. The alert will be delivered to our mailbox using Alert Notification.

Before starting, we’ll need the following:

• An active license or subscription for SAP Cloud Platform Alert Notification;
• Open source Kibana distribution enhanced with OpenDistro;
• A running application monitored with the Kibana instance.

Note: For this setup we use Kibana version 7.4.2.

Configure SAP Cloud Platform Alert Notification

This section includes both getting credentials for posting events to Alert Notification as well as instructioning the service what to do when an alert from Kibana arrives. For the sake of simplicity, we will set up the simplest action of type EMAIL. You could choose later the most convenient for you type where any issue will be immediately noticed. Further information on available action types, see here.

1. WIthin your Alert Notification instance view in the SAP Cloud Cockpit, create a technical client with BASIC authentication mechanism and ALERT_WRITE permission:
   • steps for Neo environment;
   • steps for Cloud Foundry environment.
2. Once created, you will be provided with client id and client secret – make sure to save them in a secure place, so that we could make use of them later.
3. In the Export or Import menu, paste the following configuration in the Import text field and click Import:Note: Make sure to replace the email address placeholder with one you have access to.

   {
     "conditions": [
       {
         "name": "MatchAllKibanaAlerts",
         "propertyKey": "eventType",
         "predicate": "EQUALS",
         "propertyValue": "KibanaAlert"
       }
     ],
     "actions": [
       {
         "name": "NotifyMeByEmail",
         "state": "ENABLED",
         "destination": "<your.email@here>",
         "type": "EMAIL"
       }
     ],
     "subscriptions": [
       {
         "name": "AllKibanaAlertsToMyEmail",
         "conditions": [
           "MatchAllKibanaAlerts"
         ],
         "actions": [
           "NotifyMeByEmail"
         ],
         "state": "ENABLED"
       }
     ]
   }

    

   Within several seconds, you’ll receive an email containing a confirmation code as well as a confirmation link. Make sure to confirm you email action either using the link or the confirmation code in the SAP Cloud Cockpit.

Set up Kibana

Let’s configure Kibana to monitor the number of runtime errors in our application. We will set a certain threshold and when it is met, an alert will be triggered. And where the alert is going to be delivered? To our Alert Notification instance, of course : )

If the application that is monitored by your Kibana instance does not log much errors, no worries – we could test our integration with a simulated test alert at the end.

Let’s do it:

1. In Kibana UI, navigate to the Alerting menu
2. Navigate to the Destinations tab, then select Add destination:
   • Provide meaningful name, this destination will point to Alert Notification
   • Select Custom Webhook type
   • In the Settings section, select Define endpoint by URL
   • Build and enter the Webhook URL as follows depending on youр SAP Cloud Platform environment:
     • for your Cloud Foundry Alert Notification service instance – enter the url supplied with your Service Key, followed by /cf/producer/v1/resource-events, e.g.:

       https://clm-sl-ans-live-ans-service-api.cfapps.eu10.hana.ondemand.com/cf/producer/v1/resource-events

     • for Neo accounts – use the Event Producer URL shown at Security page when you navigate to Alert Notification service in SAP Cloud Platform Cockpit, with a similar ending, e.g.:

       https://clm-sl-ans-live-ans-service-api.cfapps.eu10.hana.ondemand.com/neo/producer/v1/resource-events

   • In the Header information section, add Content-Type header with value application/json
   • Add Authorization header with value Basic <<Base64 encoded value of your credentials>>
     • the placeholder shall be replaced by the Base64 encoded representation of the string client_id:client_secret, where client_id and client_secret are the actual values we’ve received upon creation of the technical client in Alert Notification
   • Select Create
3. Navigate to the Monitors tab, then select Add Monitor:
   • Provide meaningful name – this will be the monitor for the errors in our application
   • In the Define Monitor section, select Define using visual graph (of course, if you could play with different setup later)
   • In the Index dropdown, select the index that contains the logs of your application
   • Select appropriate Time field
   • In the Create a monitor for section, input your query. For the demo we’ve decided to trace the number of “Internal Server Errors” for any 5 minute interval:
     
   • In Monitor schedule section, provide the frequency of the check – for the demo, we’ve set it up to be every 1 minute
   • Select Create
4. You’re now redirected to the Create trigger step:
   • Provide meaningful name for the trigger
   • Select severity level
   • Select trigger condition (or just replicate our demo setup)
     
     
5. In the Configure actions section:
   • Provide meaningful action name
   • From the Destionation dropdown, select the destination we created in step 2)
   • In the Message text box, provide the following template:

     {
        "eventType":"KibanaAlert",
        "resource":{
           "resourceName":"{{ctx.trigger.name}}",
           "resourceType":"cloud-resource"
        },
        "severity":"WARNING",
        "category":"ALERT",
        "subject":"Kibana Alert by {{ctx.trigger.name}} triggered",
        "body":"Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.",
        "tags":{
           "ctx.monitor.name":"{{ctx.monitor.name}}",
           "ctx.trigger.name":"{{ctx.trigger.name}}",
           "ctx.periodStart":"{{ctx.periodStart}}",
           "ctx.periodEnd":"{{ctx.periodEnd}}",
           "ctx.trigger.severity":"{{ctx.trigger.severity}}"
        }
     }​

     Note: You could later fine tune this template in accordance to your needs and preferences. It’s only required to meet the Producer API specification.

   • Select Create

Well done, we’ve just completed the setup.

Test the Setup

Let’s see how this integration will actually look like. You could do it either invoking multiple errors in your application, or just simulate it. I prefer the second option as follows: navigate back to the monitor we’ve created in step 3), then in the Triggers section, select the trigger we’ve created in step 4), then Edit. Again in the Configure actions section, expand the action created in step 5). Now hit on Send test message. Within a couple of minutes, an email like this will be in your mailbox:

Would you like to further explore this scenario? See Integrating with Kibana at SAP Help Portal.

This blog post is part of a series of blog posts related to integration between third-party monitoring tools and SAP Cloud Platform Alert Notification service. We will continuously add newly released integrations, so stay tuned!

We ask you – our customers and partners – to join us in our integration journey! Tell us what you use for monitoring solutions, what you need to integrate with them; tell us your experience, and what your alerting needs are!