SAP IoT’s Rules and Action services are core for automating business processes based on sensor data aka time series data. Since the launch of SAP IoT in March 2019 (named SAP Leonardo IoT at the time) the Rules service enables to create business insights on time series data in form of Events. These Events can then be forwarded via the Actions service e.g. to the SAP Business Suite, e.g. to trigger a replenishment process, a Service process, etc.
In the meantime, we have constantly evolved both, Rules and Action services and we are continuing the development here. This blog is more technical and will explain the enhancements of the SAP IoT Rules service from the functional point of view. Please see our other blog articles for example scenarios what insights can be realized using SAP IoT and the technical advancements of the SAP IoT Action Service.
Before giving an overview about the additional Rules features, let’s have first a view on the overworked app design of the SAP IoT Rules design time. From user studies we learnt that the rules design time app can be improved.
We have moved from a 2 apps approach with “Rules Context” and “Rules” to a 1 app design, which is called “Rule Projects”.
A Rule Project defines the data objects rules can be defined on and contains all the related rules.
When starting a new Rules project, first you give it a name, short text, and description. In the next step, you can add the data objects used for the rules. In the below screenshot you see that rules can be specified on Property Set (e.g. a rule applies to left front wheel of a vehicle) or Property Set Type level (e.g. a rule applies to all wheels of a vehicle). Further, you see that there are a few standard Data Objects that are automatically added; important to mention here are Geofence Events as well as the SAP_Stream_Derivation object, which contains the business timestamps that can be used in Derivation rules. The other objects are used as output of rules and you do not need to use them in rules definition. The rule editor automatically uses them.
From the tabs of the Rule Project app you can understand how a Rule project is structured:
In the examples used in this blog we use the following data objects:
The screenshot below shows the list of Rules define for this project. You can adapt the columns displayed to your needs. Here, you can also see for each rule the input and output data objects. This helps you to understand the rule better already from the list. And on this tab you also create new rules.
SAP IoT Rules Service leverages the SAP Cloud Platform Business Rules. If you want to understand which operators and functions are provided to define your rule, please see the Rule Expression Language 2.0 documentation.
This concludes the overview of the new Rule Projects app.
Rules used in IoT have special requirements compared to e.g. rules within business processes. IoT rules process time series data. E.g., a threshold is not only violated once. It might be violated for a longer time period. Here, creating an event for each threshold violation would just “spam” your application and nobody would care about warnings or alerts from Rules anymore. Thus, you want to receive an event only on the first occurrence of a threshold violation. For this purpose, we have introduced the Toggle mode for Streaming Rules and Scheduled Rules since over a year ago.
The Toggle mode leads to creating an event only on the first threshold violation, but not on the consecutive ones. To get a new event created, the sensor value must first return to the non-threshold violation area. This is depicted in the graphic below.
Together with the Toggle mode, we also introduced the Sleeping mode for Streaming Rules. The Sleeping Mode extends the Toggle Mode by a “cool down” period. To get a sense of urgency, you might want to receive warning and alerts again after a specific period, although the sensor value did not return to a normal state again. How this works is depicted in the below screenshot.
Since a few months we have introduced 2 additional execution modes for Streaming Rules. Both aim at avoiding false positive warnings or alerts, e.g. in the case of sensor measurement errors. Sensors do not work perfectly all the time. E.g. I use a DHT-11 temperature sensor for my Smart Home monitoring. Sometimes it sends one measurement that is totally off, e.g. 25°C, 5 seconds later -2°C, 5 seconds later again 25°C. To avoid getting an alert on the -2°C measurement you can use the Consecutive Condition Fulfillment Mode. It suppresses the first X consecutive occurrences of a “true” rule condition, here threshold violation; and only on the X+1 consecutive occurrence of “true” rule condition the rule event is created.
Similarly, you might rather be looking for a specific duration that a rule condition should be fulfilled before a rule event is created. E.g., an event shall be created only if a motor is above 80°C for 10 minutes or longer. Such scenarios you can cover using Streaming Rules with Duration mode. This is depicted below.
With all these execution modes a wide variety of IoT specific rule scenarios can be covered that create insights into the state of your Thing.
One additional comment helps you to understand the behavior of toggles, sleeping times, etc. These states are cached per Thing for 36 hours and are renewed with each rule execution – regardless of the result of the rule. So if you have a machine that continuously violates a threshold, then you turn it off during the night or stop the data transmission, and on the next morning the operations of the machine are continued violating the threshold again, the toggle will avoid that a new event is generate in the morning. No measurement not violating the threshold was received and due to the 36 hours cache, the toggle is still remembered. Toggles are only reset if there is no data ingestion for more than 36 hours – or when the rule result is “false”.
SAP IoT Rules are applied to all Things that send data for a specific Property Set or Property Set Type. That is, SAP IoT Rules addresse fleets of things. This is important to understand for scaling rules for large projects. Obviously, if you are working with larger number of things, you do not want to have to define rules per individual Thing. Here, also rules engines would not scale. Therefore, there is also a maximum number of rules you can define or activate per tenant.
However, from this you do not need to imply that all Things having the same Property Set or Property Set Type must run the exact same rules. In order to allow individual “rules” per Thing, we enable that a Rule can use the Thresholds set per Thing in the Thing Model. When you define Property Set Types for numeric data types you can also select to use up to 4 thresholds (see screenshot below). Important note: a “.” is not allowed as part of the threshold property name, if they shall be used in rules.
Defining threshold Property Sets via the API works via the so called Reference Property Set Types.
Via the Thing Modeler app or via the API you can then set individual thresholds per Thing.
When you add a Property Set or Property Set Type as Data Object to a Rule Project, the corresponding Reference Property Sets are automatically added, too. They are then available in the rule editor – currently for Streaming Rules. For the other rule types this is planned.
In order to include a meaningful rule result into your IoT scenario, the Action Service now also allows to put the thresholds into any Action message.
In conclusion: Rules with Thresholds enable you to monitor your fleet of things with only 1 rule and leveraging individual Things’ thresholds maintained in the Thing Model.
In many scenarios you want to calculate additional time series from sensor data. E.g., you want to calculate the electrical demand from voltage and amperage. Or you want to calculate the delta between the previous and the current value, or the incline by time. All this is possible also via the SAP IoT Rules Service.
For that you have the Rule type “Streaming Rules - Derivation” available.
For defining Derivation rules you need to have defined at least 1 Property Set of type Calculated Data via the Thing Properties Catalogue app or via the API (Data Category: DerivedData).
Derivation rules always as:
The screenshot below shows a derivation rule that is executed for each input measurement, as the if-condition is set to “true”.
Further you see, to calculate the incline per time, the rules leverage the previous and current measurements as well as the previous time stamp and the current timestamp.
The SAP IoT Derivation Rules provide another very important feature. Derivations handle out-of-order late comer data. Data is cached per default for 1 hour and all data coming in within this time window will be sorted and any already performed calculations of derived data will be corrected, if necessary. The maximum configurable delay time window is 5 days. For more details, please see the documentation. This feature is crucial for many scenarios, e.g. in logistics scenarios. Onboard unit cache data that cannot been sent out due to connectivity loss. When connectivity is re-established, the newest data is sent out first, because typically the most important information is the current location of a truck. Therefore, onboard units send data typically in LiFo pattern. This results in incoming data at the cloud in reverse order.
In conclusion, Derivations is a very powerful feature, especially due to the ability to use also the previous measurement and timestamp in the calculation rules. This allows e.g. to calculate the differential to a time series. Of course, you can define Streaming Rules and Schedule Rules on such derived time series.
Scheduled Rules are also a very important rule type. With Scheduled Rules you can define rules on time windows and use the aggregates provided by the SAP IoT Time Series Aggregate Store. Here, you can e.g. find out if a monthly average is above a specific threshold. Or, in order to accumulate values, you can calculate the sum of the values over a time window and compare it against a threshold. This is used e.g. in cold chain monitoring to determine, if a temperature budget was exceeded. As the timestamps of the first and last measurements of the defined time window are also available, you can also use Scheduled rules to find out if the machine has sent data e.g. in the last hour or if it went offline.
Do not underestimate the power that Schedule Rules provide to your IoT scenario to create insights!
We have documented the APIs of the SAP IoT Rules service, now. However, working with the Rules APIs is very complex. Therefore, we recommend defining your Rule Project with Data Objects and an initial rule via the UI. Based on this rule you can then adapt via the APIs.
In addition to defining and activating rules, we also offer an API to manually trigger the execution of a Scheduled Rule. In the API Scheduled Rules are called Batch Rules.
This concludes the overview about the advancements of the SAP IoT Rules service. Many additional scenarios can now be realized to create insights from time series data. This is easy to configure using our renewed Rules Project app.
Please see also our other blog that gives application examples for using SAP IoT Rules and Actions to generate insights in business scenarios. Of course, these examples leverage the available Action features.
In the meantime, we have constantly evolved both, Rules and Action services and we are continuing the development here. This blog is more technical and will explain the enhancements of the SAP IoT Rules service from the functional point of view. Please see our other blog articles for example scenarios what insights can be realized using SAP IoT and the technical advancements of the SAP IoT Action Service.
New Rules Design Time app
Before giving an overview about the additional Rules features, let’s have first a view on the overworked app design of the SAP IoT Rules design time. From user studies we learnt that the rules design time app can be improved.
We have moved from a 2 apps approach with “Rules Context” and “Rules” to a 1 app design, which is called “Rule Projects”.
A Rule Project defines the data objects rules can be defined on and contains all the related rules.
When starting a new Rules project, first you give it a name, short text, and description. In the next step, you can add the data objects used for the rules. In the below screenshot you see that rules can be specified on Property Set (e.g. a rule applies to left front wheel of a vehicle) or Property Set Type level (e.g. a rule applies to all wheels of a vehicle). Further, you see that there are a few standard Data Objects that are automatically added; important to mention here are Geofence Events as well as the SAP_Stream_Derivation object, which contains the business timestamps that can be used in Derivation rules. The other objects are used as output of rules and you do not need to use them in rules definition. The rule editor automatically uses them.
From the tabs of the Rule Project app you can understand how a Rule project is structured:
- General information
- Data Objects
- List of rules defined (using the data objects)
In the examples used in this blog we use the following data objects:
The screenshot below shows the list of Rules define for this project. You can adapt the columns displayed to your needs. Here, you can also see for each rule the input and output data objects. This helps you to understand the rule better already from the list. And on this tab you also create new rules.
SAP IoT Rules Service leverages the SAP Cloud Platform Business Rules. If you want to understand which operators and functions are provided to define your rule, please see the Rule Expression Language 2.0 documentation.
This concludes the overview of the new Rule Projects app.
New Streaming Rules Execution Modes
Rules used in IoT have special requirements compared to e.g. rules within business processes. IoT rules process time series data. E.g., a threshold is not only violated once. It might be violated for a longer time period. Here, creating an event for each threshold violation would just “spam” your application and nobody would care about warnings or alerts from Rules anymore. Thus, you want to receive an event only on the first occurrence of a threshold violation. For this purpose, we have introduced the Toggle mode for Streaming Rules and Scheduled Rules since over a year ago.
The Toggle mode leads to creating an event only on the first threshold violation, but not on the consecutive ones. To get a new event created, the sensor value must first return to the non-threshold violation area. This is depicted in the graphic below.
Together with the Toggle mode, we also introduced the Sleeping mode for Streaming Rules. The Sleeping Mode extends the Toggle Mode by a “cool down” period. To get a sense of urgency, you might want to receive warning and alerts again after a specific period, although the sensor value did not return to a normal state again. How this works is depicted in the below screenshot.
Since a few months we have introduced 2 additional execution modes for Streaming Rules. Both aim at avoiding false positive warnings or alerts, e.g. in the case of sensor measurement errors. Sensors do not work perfectly all the time. E.g. I use a DHT-11 temperature sensor for my Smart Home monitoring. Sometimes it sends one measurement that is totally off, e.g. 25°C, 5 seconds later -2°C, 5 seconds later again 25°C. To avoid getting an alert on the -2°C measurement you can use the Consecutive Condition Fulfillment Mode. It suppresses the first X consecutive occurrences of a “true” rule condition, here threshold violation; and only on the X+1 consecutive occurrence of “true” rule condition the rule event is created.
Similarly, you might rather be looking for a specific duration that a rule condition should be fulfilled before a rule event is created. E.g., an event shall be created only if a motor is above 80°C for 10 minutes or longer. Such scenarios you can cover using Streaming Rules with Duration mode. This is depicted below.
With all these execution modes a wide variety of IoT specific rule scenarios can be covered that create insights into the state of your Thing.
One additional comment helps you to understand the behavior of toggles, sleeping times, etc. These states are cached per Thing for 36 hours and are renewed with each rule execution – regardless of the result of the rule. So if you have a machine that continuously violates a threshold, then you turn it off during the night or stop the data transmission, and on the next morning the operations of the machine are continued violating the threshold again, the toggle will avoid that a new event is generate in the morning. No measurement not violating the threshold was received and due to the 36 hours cache, the toggle is still remembered. Toggles are only reset if there is no data ingestion for more than 36 hours – or when the rule result is “false”.
Rules with Thresholds
SAP IoT Rules are applied to all Things that send data for a specific Property Set or Property Set Type. That is, SAP IoT Rules addresse fleets of things. This is important to understand for scaling rules for large projects. Obviously, if you are working with larger number of things, you do not want to have to define rules per individual Thing. Here, also rules engines would not scale. Therefore, there is also a maximum number of rules you can define or activate per tenant.
However, from this you do not need to imply that all Things having the same Property Set or Property Set Type must run the exact same rules. In order to allow individual “rules” per Thing, we enable that a Rule can use the Thresholds set per Thing in the Thing Model. When you define Property Set Types for numeric data types you can also select to use up to 4 thresholds (see screenshot below). Important note: a “.” is not allowed as part of the threshold property name, if they shall be used in rules.
Defining threshold Property Sets via the API works via the so called Reference Property Set Types.
Via the Thing Modeler app or via the API you can then set individual thresholds per Thing.
When you add a Property Set or Property Set Type as Data Object to a Rule Project, the corresponding Reference Property Sets are automatically added, too. They are then available in the rule editor – currently for Streaming Rules. For the other rule types this is planned.
In order to include a meaningful rule result into your IoT scenario, the Action Service now also allows to put the thresholds into any Action message.
In conclusion: Rules with Thresholds enable you to monitor your fleet of things with only 1 rule and leveraging individual Things’ thresholds maintained in the Thing Model.
Calculated Time Series aka Derivations
In many scenarios you want to calculate additional time series from sensor data. E.g., you want to calculate the electrical demand from voltage and amperage. Or you want to calculate the delta between the previous and the current value, or the incline by time. All this is possible also via the SAP IoT Rules Service.
For that you have the Rule type “Streaming Rules - Derivation” available.
For defining Derivation rules you need to have defined at least 1 Property Set of type Calculated Data via the Thing Properties Catalogue app or via the API (Data Category: DerivedData).
Derivation rules always as:
- Input: 1 Property Set of type Measured Data (in the API: Data Category: TimeSeriesData)
- Output: 1 Property Set of type Calculated Data (in the API: Data Category: DerivedData)
The screenshot below shows a derivation rule that is executed for each input measurement, as the if-condition is set to “true”.
Further you see, to calculate the incline per time, the rules leverage the previous and current measurements as well as the previous time stamp and the current timestamp.
The SAP IoT Derivation Rules provide another very important feature. Derivations handle out-of-order late comer data. Data is cached per default for 1 hour and all data coming in within this time window will be sorted and any already performed calculations of derived data will be corrected, if necessary. The maximum configurable delay time window is 5 days. For more details, please see the documentation. This feature is crucial for many scenarios, e.g. in logistics scenarios. Onboard unit cache data that cannot been sent out due to connectivity loss. When connectivity is re-established, the newest data is sent out first, because typically the most important information is the current location of a truck. Therefore, onboard units send data typically in LiFo pattern. This results in incoming data at the cloud in reverse order.
In conclusion, Derivations is a very powerful feature, especially due to the ability to use also the previous measurement and timestamp in the calculation rules. This allows e.g. to calculate the differential to a time series. Of course, you can define Streaming Rules and Schedule Rules on such derived time series.
Scheduled Rules
Scheduled Rules are also a very important rule type. With Scheduled Rules you can define rules on time windows and use the aggregates provided by the SAP IoT Time Series Aggregate Store. Here, you can e.g. find out if a monthly average is above a specific threshold. Or, in order to accumulate values, you can calculate the sum of the values over a time window and compare it against a threshold. This is used e.g. in cold chain monitoring to determine, if a temperature budget was exceeded. As the timestamps of the first and last measurements of the defined time window are also available, you can also use Scheduled rules to find out if the machine has sent data e.g. in the last hour or if it went offline.
Do not underestimate the power that Schedule Rules provide to your IoT scenario to create insights!
APIs
We have documented the APIs of the SAP IoT Rules service, now. However, working with the Rules APIs is very complex. Therefore, we recommend defining your Rule Project with Data Objects and an initial rule via the UI. Based on this rule you can then adapt via the APIs.
In addition to defining and activating rules, we also offer an API to manually trigger the execution of a Scheduled Rule. In the API Scheduled Rules are called Batch Rules.
This concludes the overview about the advancements of the SAP IoT Rules service. Many additional scenarios can now be realized to create insights from time series data. This is easy to configure using our renewed Rules Project app.
Please see also our other blog that gives application examples for using SAP IoT Rules and Actions to generate insights in business scenarios. Of course, these examples leverage the available Action features.
- SAP Managed Tags:
4 Comments
| User | Count |
|---|---|
| 8 | |
| 8 | |
| 8 | |
| 4 | |
| 3 | |
| 3 | |
| 3 | |
| 3 | |
| 2 | |
| 2 |
