Best Practices for using current SAP APM Rules for Condition Monitoring

[marcus_behrens]

Consider ingesting only relevant changes of indicator values to reduce costs and increase granularity!

In principle there are two common strategies for determining which data to send to condition monitoring applications like APM. One strategy is to ingest for all indicators the latest value (or the latest average value) in regular intervals, e.g. every minute, every hour or every day. The alternative is to ingest indicator values whenever the value has changed (e.g. temperature went up by 1 °C) taking into consideration whatever precision is sufficient for condition monitoring or any other APM use case (e.g. some use cases may require tracking temperature changes with a precision of 0.1 °C instead 1 °C).

Example: 

Regular ingestion of a temperature (measured in °C) and a relative humidity (measured in %) indicator:

1. 09:00 UTC: 15 °C, 34% (first values reported for these indicators)
2. 10:00 UTC: 15 °C, 34%
3. 11:00 UTC: 15 °C, 35%
4. 12:00 UTC: 16 °C, 35%
5. ...

"On Change" ingestion:

1. 09:00 UTC: 15 °C, 34% (first values reported for these indicators)
2. 10:34 UTC:            35%
3. 11:55 UTC: 16 °C
4. ...

Benefits of regular interval based approach:

1. Easier for applying certain data processing and visualization techniques which assume indicator value timestamps have equal/uniform distances
2. Possibility to detect missing data points or disrupted data integration flows easily, because of predictable sampling frequency/timing behavior per indicator

Benefits of “on change” approach:

1. Reduced amount of consumed data streams in case there are longer “pauses” between relevant changes of indicator values → less data traffic and cloud streaming costs
2. Even if overall data resolution might be lower compared to regular interval based approach, relevant value changes can be tracked (even more) precisely based on their exact change timestamps

Issues of "on change" approach:

1. Time series data points for different indicators might have unaligned/non-uniform time stamps which can make applying aggregation or other algorithms on raw data tricky
2. Batching or packaging of values can be harder, but is still possible when using timestamps indicating when the physical measurement/event happened ("business timestamp" - see below)
3. Detecting potentially missing time series data points becomes more difficult, because an indicator does not have an inherent and predictable sampling frequency/timing behavior

Even a mix of both strategies is possible, for example to regularly ingest each indicator once per hour or per day and additionally ingest relevant changes. This gives you the benefit of a "regular heartbeat" for all indicators but at the same time mitigates some of the disadvantages of the "on change" approach.

Package indicator values into batches across positions, categories and technical objects!

There are 2 different channels for ingesting data into APM from device to cloud or cloud to cloud:

1. Via an MQTT/REST endpoint provided by SAP IoT, directly from device or from router device (see https://help.sap.com/docs/SAP_IoT/226d46a15bb245b7bf8126604bd6f0fb/755de2516dde4fafb446efaaafb2c81a....)
2. Via the APM “measurements” API (see https://api.sap.com/api/Timeseries_APIs/resource/Batch_Requests)

The first option offers the possibility to ingest values for multiple indicators, multiple positions or even multiple categories for one technical object. This means you can for example send ten values for ten different indicators that are in one position, in one category in one technical object for one timestamp at the same time. The benefits of this kind of “batching” at the position level are the following:

1. The number of data streams metered relative to your SAP contract can be reduced as multiple data points still count as one data stream
2. Streaming rules, applicable only to indicators in one position execute only one time instead of multiple times
3. The end-to-end performance of the ingestion pipeline and rules processing can be improved – batch processing allows for higher throughput then message by message processing

There are also down-sides to this approach:

1. Packaging multiple measurements requires your device or your gateway to buffer and bring together multiple records

To review some examples for different types of batching please refer to the aforementioned documentation.

Use business timestamp whenever you can but also ingest steadily and avoid delays!

If you do not provide a timestamp for an indicator value that you ingest, APM will use the current UTC timestamp at the time of receiving the message as the “business timestamp”. So if you measured the value five minutes ago and it took two minutes to transmit the message, the timestamp recorded and used by APM will be seven minutes later than the actual measurement/event. To avoid this, APM applications are currently designed to deal with business timestamps up to 15 minutes into the future and up to three months (retention time in warm store) into the past. The current implementation of APM rules will also recognize this order, but values asserted or alerts generated from rules will have the timestamp from processing. Additionally, when a rule is triggered, whatever is known at this point in time, will be considered when evaluating the rule. So it might be that only the last time the rule fires, is the time where it applies the condition or calculates the value correctly.

For examples for using or omitting the business timestamp please refer to the aforementioned documentation.

Be aware of what SAP monitors and what you have to monitor!

SAP will monitor that all rules are executed in general in all tenants. The logic of the rule condition defined by you is evaluated by SAP when the rule executes and we do not check if the logic result produced is what you expect in all conditions. This is something you have to test and monitor to be reliable depending on what you expect. In any case when the rule is applied, the timestamp of last execution is updated - both when the condition evaluates to true or to false. The same is the case for the consequences of the rule. We will make sure the alert is created or the calculated value is written, but we have no way of knowing if it is the correct value. So also here testing and monitoring (e.g. sampling) might be required by the customer.

The ingestion path before an ingested value reaches SAP APM application (either via SAP IoT endpoint or APM API) is typically built up as part of a customer implementation project and has to be monitored by the customer. The clients side calling MQTT or HTTP endpoints have to handle situations in which the endpoint might not be available or returning error codes like 403, 404, 429. SAP will routinely monitor and pick up e.g. 500 errors as part of their operations effort with given priority. If you encounter any high severity issues because of the described behavior, please create a customer incident appropriate priority to get support from SAP.

Use scheduled rules instead of streaming rules!

Scheduled rules and streaming rules offer both the ability to trigger a notification or an alert. But there are benefits to using scheduled rules:

1. Scheduled rules have a clear and point in time when they are triggered and they run exactly once. So the business timestamp generated from the scheduled rule is clearly defined.
2. Scheduled rules are more performant then streaming rules as they “wait” for the data to be there and then apply themselves in a controlled and deterministic way.
3. Scheduled rules allow you to define execution modes that allow to control how often they are triggered again.

Streaming rules have these advantage:

1. If you are not sure which indicator will come in at which time then the streaming rule will provide the most immediate reaction.

Examples of scheduled rules can be found at https://help.sap.com/docs/SAP_APM/1bb12075258a41e1a024d28a6ddfe246/c63996a32f2a47d5bba4c108870caeaa.....

Do not use rules on indicators changing more often then once a minute!

This recommendation has been added to the APM documentation in November 2023 here: https://help.sap.com/docs/SAP_APM/1bb12075258a41e1a024d28a6ddfe246/e93178793bb04ad98432419c8a6dc745..... The reason is that for all types of rules the execution of the rule (which takes up to 60 seconds but in most cases a few seconds) can overlap with the next value coming in and this can make the rule fire incorrectly or generate an incorrect result. As long as all indicators used by a rule are not changing more often then once a minute, you should be good. For scheduled rules, this means you should make sure that the values it is based on are "in" before the rule runs. For the other rule types it means they simply might not work correctly, when applied to an indicator value that changes more often then every 60 seconds.

Be careful when applying a calculation or streaming rule to multiple indicators!

If you cannot use scheduled rules, be aware that chained rules or streaming rules based ontwo or more indicators will execute for each value that is coming in. This means that also the consequence (alert, calculated value) is happening multiple times. This might be an issue from a performance point of view and also from a logic point of view. To prevent this you might want to use references to timestamp differences in the condition of the rule.

Example: For the rule a+b=c the following happens to c as a and b come in:

1. the values of a and b are 0
2. the value a and b come in with the same business timestamp at about the same time with a=1 and b=2
3. dependent on which one is processed first, the value of c could temporarily become 1 (1+0) or 2 (0+2)
4. the second time the rule fires, c should become 3

Do not use loops between calculation rules!

For intermediate calculations, scheduled rules cannot be used. But calculation rules have additional pitfalls. The APM rules documentation explains how to workaround the fact that calculation rules prevent you from using the same indicator as both output and input (see https://help.sap.com/docs/SAP_APM/1bb12075258a41e1a024d28a6ddfe246/14ca0a786f394b9db6e9938339cf2832....). This is to prevent loops. Please do not use this workaround if you are not aware of the implications as it can make the rules engine spin in circles. There can also be significant data stream usage generated based on values being ingested very frequently and this will create avoidable costs and a huge amount of unneeded data in your APM tenant's database.