Technology Blogs by SAP
Learn how to extend and personalize SAP applications. Follow the SAP technology blog for insights into SAP BTP, ABAP, SAP Analytics Cloud, SAP HANA, and more.
cancel
Showing results for 
Search instead for 
Did you mean: 
stojanm
Product and Topic Expert
Product and Topic Expert
The topic of data sovereignty is omnipresent. Enterprises and public bodies are gradually becoming reliant on digital services throughout their whole value chain and operations and the pace is accelerating due to various disruption sources. In addition, the amounts of data being generated, analyzed and shared is increasing exponentially. It is for those two reasons that the question of controlling data distribution, usage and storage becomes extremely important.

The European initiative - GAIA-X - addresses those points by defining an open, transparent digital ecosystem, where data and services can be made available, collated and shared in an environment of trust. It aims to provide data-driven business models with a tool to overcome the main challenge: the lack of a scalable data governance for federated data stored on any platform.

SAP belongs to the founding members of GAIA-X and is also a member of the International Data Space Association IDSA that strives to create the future of a global, digital economy with International Data Spaces (IDS), a secure, sovereign system of data exchange in which all participants can realize the full value of their data.

In this blog post I will discuss the results of a proof of concept (PoC), in which services from SAP Business Technology Platform (SAP BTP) were used to realize a data usage control as defined in the IDS reference architecture. The usage control is the heart of the data sovereignty concept, since it provides the technical means for data owners to control the terms and conditions under which their data is used by data consumers.

SAP BTP as a technology provider for the usage control framework offers a rich set of technical features - e.g. several runtime environments (e.g. managed Kubernetes, etc.) and a range of tools suited for citizen as well as full stack developers (e.g. low-code / no-code, etc.). In addition to the solid technological foundation, BTP exposes services aiming to assist in the whole Data-to-Value process. Those services can be summarized in the following three groups:

  • Connectivity: Out-of-the-box services for connectivity and data processing / modeling;

  • AI/ML: Tools for advanced analytics, Machine Learning and AI, which allow the extraction of insights from data;

  • End-to-end integration: Services allowing to forward the extracted insights to the Intelligent Suite and to integrate those into Business Processes, thus accelerating the actual value generation;


Quick intro on data usage control within the IDS connector


The meaning of access control is intuitively understood by most users of IT systems nowadays. It is the concept, which defines who is allowed to perform certain actions on resources and documents. While this is sufficient in traditional IT, in the context of data access and processing, it is necessary to attach additional rules and restrictions to the usage (e.g. user is not allowed to use the data for analytics or only high granularity analysis is allowed, etc.). This defines the purpose of a data usage control - a data-centric view on restrictions framework, which allows data producers to define, negotiate and attach a set of rules (policies) for the usage of their data.

Within the IDS concept, the task of definition, negotiation and enforcement of usage-related policies is taken over by the usage control, which is an essential part of the IDS connector. In the figure below a schematic view on the main technical components involved in IDS is presented. On a high level the communication process between the involved parties is as follows: the data provider advertises its data to an IDS broker, which can be queried by data consumers in the data discovery phase. Once a data provider has been discovered by the consumer, the communication happens in a peer-to-peer manner using the IDS connectors, which have the task not only to transfer the data, but also to negotiate and enforce the data usage policies. The IDS usage control is responsible for IDS policy enforcement.


High level view on the main components of IDS as defined by the reference architecture


When it comes to how the data policies are handled by the usage control, the IDS reference architecture model foresees the logical separation in different entities. For example, the policy enforcement point (PEP) is responsible for interception of data requests and forwarding of the requests to a policy decision point (PDP) for decision making. If the policies involved define a data transformation operation (e.g. data needs to be anonymized before usage, etc.) this task is logically executed by the policy execution point (PXP).



Usage Control framework within the IDS Connector as defined by the IDS RAM Version 3


From the image above, the Policy Decision Point (PDP) is the element within usage control responsible for the technical enforcement of the usage control policy and provision of a decision back to the PEP and the IDS Connector. In the following section I will present how API Management, which is part of the Integration Suite in the SAP Business Technology Platform, was utilized to realize an IDS Policy Decision Point (PDP) for policy enforcement using a low-code approach.

SAP BTP-based usage control - high level idea


SAP API Management (APIM) allows customers to define API proxies of existing APIs, thus not having to change the actual API implementation. No modifications on already defined endpoints are required, while anything related to traffic management, security and consumption is taken care by the service using the concept of policies. This means that only the proxy API is exposed to users and the API creator defines a chain of checks and rules, which need to be completed before the API request is forwarded to the endpoint. These chains are called APIM flows and look as shown below:


An example API Management flow, showing the concept of a proxy API, including the definition of several security and traffic policies.


The service offers a rich collection of APIM policies, which are available out-of-the-box and which allow the user to implement APIM flows and define access logic. The following figure shows a selection of the APIM policies grouped per high level topic. As it can be seen, there are policies available for type conversion (e.g. JSON to XML), for security enforcement tasks (e.g. OAuth v2.0), for traffic management (e.g. Spike Arrest) and also for extensibility (e.g. Python Script).


Selected policies built-in in API Management


The features of API Management, which make it well suited for use in the context of IDS Usage control can be summarized as follows:

  • Implementable as templates: APIM flows can be stored as templates and be shared with other users of the platform;

  • Can be generated programmatically: The APIM policy flows can be designed per drag and drop, but also programmatically.  The flows are internally represented as XML files, which get parsed by the engine during policy enforcement;

  • Extendable via Python and JavaScript: To implement functionalities, which are not already available out-of-the-box;

  • Can trigger external APIs: Operators are provided, which allow to make HTTP/S calls to internal / external services;


SAP BTP-based usage control - an example


In this section, an IDS standard policy will be implemented using SAP's API Management, as an example to provide a better understanding of the processes involved.

The "restricted number of usages" IDS policy allows the data provider to define an upper limit for the data usage within a certain time period. IDS policies are defined using Open Digital Rights Language (ODRL) and the JSON configuration for the selected IDS policy is:
"ids:permission": {
"ids:target": {
"@id":"?artifactURI"
},
"ids:action": "idsc:USE" ,
"ids:constraint": [{
"ids:leftOperand":{"@id":"idsc:COUNT"},
"ids:operator":{"@id":"idsc:LTEQ" },
"ids:rightOperand": {
"@value": "?quantity",
"@type": "xsd:decimal"
},
"ids:pipEndpoint":{"@id":"?counterPipURI"}
}],
"ids:postDuty": [{
"@type": "ids:Duty",
"ids:action": [{
"rdf:value": { "@id": "idsc:INCREMENTCOUNTER" }
}],
"ids:PXPEndPoint":{"@id":"?increaseCounterPXPUri"}
}]
}

The components of the policy definition, which are relevant for our case are: the name of the policy and the processing logic of the policy itself (e.g. here the IDS policy defines that the number of usages idsc:COUNT has to be less-than-or-equal to idsc:LTEQ for the data request to be allowed).

To decode and enforce this IDS policy, the following API Management flow was designed: 

An example API Management flow to parse and enforce the Restricted Number of Usages policy from the IDS Standard


The process is triggered when the Policy Enforcement Point (PEP) forwards the JSON representation of the IDS policy and asks the Policy Decision Point (PDP), here the API Management flow shown above, about a decision regarding the current data request. The APIM flow processes the request through the following three operators (left to right) in order to return the decision back to the requestor (PEP):

  • idsReqDispatcher: A parser, which decodes the JSON of the attached IDS policy, extracts the values and assigns them to variables that are visible for all operators of the APIM flow;

  • numUsage: An out-of-the-box policy enforcer, which accepts as an input parameter the number of usages allowed, keeps track on the number of usages at any given time and throws an exception if the value is exceeded;

  • idsError: A request handler, which generates the error message in case of limit exceeded event. Based on the outcome of the previous operator, a decision of True or False is sent back to the Policy Enforcement Point (PEP);


The following snippets include the generated XML configuration for each of the three steps of the APIM flow. The main XML document shows the sequence of steps encoded into the <step/> section:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ProxyEndPoint default="true">
<preFlow>
<name>PreFlow</name>
<request>
<isRequest>true</isRequest>
<steps>
<step>
<policy_name>idsReqDispatcher</policy_name>
<sequence>1</sequence>
</step>
<step>
<policy_name>numUsage</policy_name>
<sequence>2</sequence>
</step>
<step>
<policy_name>idsError</policy_name>
<condition>ratelimit.numUsage.failed=true</condition>
<sequence>3</sequence>
</step>
</steps>
</request>
</preFlow>
</ProxyEndPoint>

The parser (idsReqDispatcher) XML configuration is shown below. The operator parses the JSON structure of the incoming IDS policy and assigns the extracted values to a variable structure called myresp. Those values are visible for all elements of the main APIM flow (e.g. left_OP: describes the type of policy and myCNT: the number of usages allowed).
<ExtractVariables async="true" continueOnError="false" enabled="true" xmlns='http://www.sap.com/apimgmt'>
<Source>request</Source>
<VariablePrefix>myresp</VariablePrefix>
<JSONPayload>
<Variable name="left_OP">
<JSONPath>$.ids:permission.ids:constraint[0].ids:leftOperand.@id</JSONPath>
</Variable>
<Variable name="myCNT">
<JSONPath>$.ids:permission.ids:constraint[0].ids:rightOperand.@value</JSONPath>
</Variable>
<Variable name="operation">
<JSONPath>$.ids:permission.ids:constraint[0].ids:operator.@id</JSONPath>
</Variable>
</JSONPayload>
</ExtractVariables>

The next element of the flow is the policy enforcer (numUsage). It uses the local variables extracted during the previous step to determine if the type of IDS policy is as expected (in our case restricted number of usages) and forwards the value of mycnt to the standard operator. The standard operator is now set up to count the number of calls in a given time period (here <TimeUnit>1 minute</TimeUnit>) and will return the results of the check to the succeeding element of the APIM flow:
<Quota async="false" continueOnError="true" enabled="true" type="calendar" xmlns="http://www.sap.com/apimgmt">
<Allow countRef = "myresp.myCNT" />
<Interval>1</Interval>
<Distributed>true</Distributed>
<StartTime>2015-2-11 12:00:00</StartTime>
<Synchronous>true</Synchronous>
<TimeUnit>minute</TimeUnit>
</Quota>

Finally, the exception handler (idsError) checks the output of the policy enforcement and decides on the return values (in this case either 200 if the limit is not exceeded otherwise code 429). And these response codes can be converted into a True or False value respectively:
<RaiseFault async="true" continueOnError="false" enabled="true" xmlns="http://www.sap.com/apimgmt">
<FaultResponse>
<Set>
<Headers>
<Header name="Retry-After">{ratelimit.numUsage.expiry.time}</Header>
</Headers>
<Payload contentType="text/plain">Your quota exceeded </Payload>
<StatusCode>429</StatusCode>
<ReasonPhrase>Too many Requests</ReasonPhrase>
</Set>
</FaultResponse>
</RaiseFault>

The big picture


It is important to emphasize, that the presented concept is modular and scalable. For example, oftentimes IDS policies are chained sequentially to realize a more complex usage control logic. Using APIM flows, a generic IDS policy parser can be built, which is able to recognize, parse and enforce such IDS policies. And if some functionality is not available out-of-the-box, the Python and JavaScript operators can be used to extend. Furthermore, such a generic parser can be stored as a template and made available as a ready-to-use content on the SAP BTP platform.

To put the concept presented here into a broader context, the following figure shows an example architecture for the realization of all components of the usage control: PEP, PDP and PXP based on BTP services:


An example high level architecture of a PDP, PEP and PXP mapped to SAP BTP services


In this blog post the PDP part of the connector was shown, handling the enforcement of IDS policies. When more complex processes are involved, e.g. data modification or even deletion of data - the PXP comes into play. For those processes a conceptual solution was designed using the Cloud Platform Integration service from the SAP Integration Suite. This solution is not part of the scope of the current blog post.

Outlook


The example described in this blog post demonstrated how SAP API Management, which is part of SAP Business Technology Platform, can be used to realize an IDS usage control for IDS policy execution and enforcement. This fact, combined with the existing capabilities of SAP BTP for data processing, analytics and AI, as well as the integration into any other LoB application, extend the possibilities available for the realization of new data-driven business models, which will increasing rely on data governance and sovereignty. Examples are pay-per-use of assets, transport networks dependent on distributed AI, as well as any use case related to sustainability, traceability and circular economy.

I would like to thank nadine.gaertner and team for the support with this blog post and all topics around IDS.

Please reach out to me with any questions and comments on the topic - I would be more than happy to discuss. Cheers, Stojan
2 Comments