What are alerts and how do they work in SAP IT Operations Analytics?

An alert in SAP IT Operations Analytics (ITOA) is an object that describes the (repeated) occurrence of an event or situation. While ITOA continuously acquires data, it also allows for automated analysis whether the incoming data or changed database states yield a certain situation one would like to be alerted about, so that one can react on those situations. An alert in ITOA consists of mainly two parts:

• The condition that allows for identifying a singular situation
• The action you would like to take in response to the situation

In addition to those, there are a few more, optional parameters that can be configured:

• The configuration object specifies the object according to which the alert should be issued. E.g. if one would like to be alerted about irregular memory consumption of servers, the configuration object will be the attribute that holds the identifier for servers.
• The trigger time specifies at what intervals ITOA scans for this alert. If no trigger time is specified, ITOA will run this scan every minute.
• The suppression time defines that the action is only performed at the certain intervals and suppressed for the length of such an interval in order to avoid alert storms. The job that identifies situations according to the previously defined trigger time will still identify the situation and increase the counter, but the corresponding action will not be triggered. If no suppression time is defined, the trigger time determines when the action is performed.

Condition

Unlike many other tools, ITOA does not only check for occurrences of events that match a specific filter, but it allows for defining complex conditions checking for specific states in the database. The Analytics Builder component within ITOA and its “guided SQL framework” allows for defining conditions by drawing on SQL and almost all of its vocabulary. The logic is that a select statement is executed to check if there are any alerts to be created. If this select statement does not return any data tuples, no alert will be generated. Otherwise, there will be an alert for each of the returned data tuples.

While this might come across a bit complex and maybe even complicated at first, the full potential should become obvious when we consider the following abstract scenario:

Event 1 creates a database state A, and event 2 creates database state B. However, if event 2 follows event 1 – and only then – the two events create database state C.

With the SQL power of alert conditions for ITOA, you can easily check for the database state C rather than checking for the occurrence of event 2 after event 1.

In addition, one can easily draw on aggregate functions like count, sum, average, round etc.

Actions

As of SAP IT Operations Analytics 2.0 SP03, ITOA allows for two different types of actions:

1. Email notifications: Send out notification emails upon identification of a specific situation.
2. Script execution: Trigger the execution of a script on the operating system-level of one of the connected SAP Host Agent hosts.

The second option is an extremely powerful way to automated IT and data center operations by taking self-healing measurers automatically without the involvement of and potential delay by humans. Examples for such automated actions can be the creation of a ticket in your service management system of choice by calling their APIs or triggering actions in a server, network or virtualization management suite to e.g. programmatically allocate more memory to a VM that is about to run into an OOM error.

Tutorial

This tutorial was built using the data set that comes with the Cloud Appliance Library (CAL) image for SAP IT Operations Analytics 2.0 SP03. You can follow this tutorial also by drawing on your own data set and adapting the steps accordingly.

Scenario

Let’s say we would like to define an alert for the following requirements:

• We would like to be notified about a server that has had less than 20% of free memory in average within the last 10 minutes.
• We would like to receive an alert per server.
• We would like to include master data in the alert notification about the servers stakeholder, SLA priority, business owner and support team.

Note: The events that might be indicative of this situation as well as the master data attributes are available in the demo data set that comes with the CAL image for ITOA. In case you are working with your own data set, you need to adapt the subsequent steps accordingly.

Step 0: Creating a new story with Analytics Builder

The preferred way to define alerts in ITOA is to go via the Analytics Builder. So, navigate to the Analytics Builder within your ITOA system and create a new story.

Step 1: Defining the first part of the condition

Within the story, you can mix other Analytics Builder artifacts like UI elements for charts etc. with alert definitions, but our recommendation would be to keep them separate for the sake of overview and transparency.

Your blocks of choice for defining alerts in Analytics Builder can be found in the Alert section of the palette. The overarching block for alert definitions is the Alert block. Drag and drop one on the canvas. Immediately, you can give this alert a Name, a Description, and a Severity. Please note that the name and the description which are specified here will later be visible in the alert management dashboard. The name of the story is just the name of the “development object” in the Analytics Builder. Please also note that the severity of alerts needs to be set to a literal by choosing a value from the drop-down list. With the current version, this cannot yet be defined dynamically by binding it to another value.

Let’s now define the first part of the condition when this alert should actually trigger. Therefore, please connect a Select block to the alert’s Data socket. To add the required Where, Group By and Column sockets to your Select block, click on the little gear-wheel icon of the Select block, tick the Filter checkbox for the Where socket and drag and drop the Columns for the other sockets as needed.

In our scenario, the event data to check for the memory consumption of hosts sits in the following table:

ITOA_ADMIN_ALERTS_PERFORMANCE.EVENT_ATTRIBUTE

So, let’s connect a Table block to the From socket of the Select block and optionally wrap an alias around it, which we can call “evt”. In the Table block select the aforementioned schema and table.

As we would like to know the host name and the rounded, average, free memory, we need to add those as Columns to the Select block. For the free memory, we need to wrap this column in an average and optionally in a round function block. As we want to have the free memory grouped by host, we also need to add the host name column to the Group By socket of the Select block.

The first part of the condition we are focusing on now is to reduce the result set to the events that occurred during the last 10 minutes only. Remember, our scenario was to only be alerted if the average free memory of the last 10 minutes dropped below 20%. Hence, let’s add a Not older than block from the SQL-Function section in the palette to the Where socket and configure it like this:

evt.TIMESTAMP not older than 10 Minutes

Your alert definition should now look like this:

Step 2: Defining the second part of the condition

The second part of our condition will now be to reduce the data tuples to only those that match the condition that the average free memory was less than 20%. This value is the result of the aggregate function of the select statement we have just defined. To apply another condition on this, we basically need to make this previous select statement an inner select and then add another where-clause on the outer select.

To do so, please wrap the previous Select block in another Alias block for tables and call it e.g. “trigger”, add another Select block to the Data socket of the Alert block, and connect this “trigger” Alias block wrapping the previous Select block to the From socket.

To the Where socket of the new Select block, we can now add the following logical expression:

trigger.memory < 20

You can find the required logical block in the Logic section of the palette, and the block for numbers in the Math section.

Your alert definition should now look like this:

Step 3: Joining master data with the event data

This next step is optional in theory, but will complement this tutorial by showing how easy it is to join master data to events in order to enrich the event messages and notifications.

In our scenario, the master data about servers, their stakeholders, SLAs, business owners etc. sits in the following table:

SAP_ITOA_MD.MASTERDATA

To join this table, add the Join socket to the outer Select block and add an on block to it. You can find this on block in the SQL-Column section of the palette. The first socket of this on block expects a Table block or an Alias block wrapping a table, and the second socket expects the logical condition specifying on which columns these two tables need to be joined. Assuming that we join the aforementioned master data table using an alias “mst”, the logical condition would look like this:

mst.HOSTNAME = trigger.hostname

Your alert definition should now look like this:

Step 4: Defining the return values, alert message and other configuration attributes

As a next step we can define which attributes from the data tuples we would like to pass on, so that they can be used within the alert message and actions. In our scenario we are going to return the following columns:

mst.STAKEHOLDER as STAKEHOLDER
trigger.hostname as INSTANCE
mst.SLA_PRIORITY as SLA PRIORITY
trigger.memory as MEMORY
mst.BUSINESS_OWNER as BUSINESS OWNER
mst.ADMIN_GROUP as SUPPORT TEAM

Your alert definition should now look like this:

To define the alert message, we need to drag a “Free Text” block from the Alert section of the palette and connect it to the Message socket of the Alert block. The “Free Text” block works like this: By clicking on the question mark icon, you can open a text area in which you can key in any text you like. To add parameters to the text, you can define these parameters inline by escaping the parameter names with double curly brackets. As soon as one of these parameters is identified, the Analytics Builder will dynamically add another socket to this block, so that you can bind values to this parameter.

In our case let’s add the following text to the alert message:

HIGH MEMORY UTILIZATION [{{current_value}}% Free]

As soon as you entered this text, there will be a new socket called ‘current value’ is: to which you can now add a String binding block and draw on the “MEMORY” column returned by the previous select statement.

Similarly, you can use a String binding block to define the configuration object to be “INSTANCE”. Doing so, you define that new alert occurrences for the same instance (i.e. hostname) will simply increase the counter of that alert.

The Trigger Time and Suppression Time sockets can both be filled drawing on the Time blocks from the Alert section of the palette. In our scenario, it makes most sense to set both these times to 10 minutes, as we are only checking for events within the last 10 minutes.

Your alert definition should now look like this:

Step 5: Defining the email notification as an action

As a last step, we are now going to define to send email notifications as an action. To do so, drag a Send Notifications block from the Alert section and connect it to the Action(s) socket.

You can fill all the sockets of the Send Notifications block using “Free Text” blocks or Variables in case you would like to be more dynamic. In our case, we will keep it simple and will only use the former. The usage of these blocks is the same as before: define the text in the text area after clicking on the question mark icon, define variables by escaping double curly brackets, bind values to these variables by using String binding or Timestamp binding blocks.

Here are the exemplary texts for Subject and Message used in this tutorial:

High Memory Utilization of only {{memory}}% Free Detected for {{hostname}}

High memory utilization of only {{memory}}% free detected for {{hostname}}.
Support team: {{supportteam}}
Stakeholder: {{stakeholder}}
SLA Priority: {{slapriority}}

As before, you can bind the corresponding values to the variables. Your alert definition should now look like this:

After saving your changes in Analytics Builder, you should see the first alert instances in the Alert Management dashboard and receive email notifications – provided there are new events that satisfy your alert condition.

Appendix

Configuring SMTP for Email Notifications

One prerequisite for making your email notifications work is that you have configured SMTP in your SAP HANA XS configuration. Please have look at the official SAP HANA Administration Guide.