Splunk – Part 1 : SAP APIM Logging & Monitoring

The OWASP Foundation’s API Security Project lists Insufficient Logging & Monitoring as one of the topmost critical API security Risks. Insufficient logging and monitoring, coupled with missing or ineffective integration with incident response, allows attackers to take advantage and fully compromise systems. So it’s important to have the right logging and monitoring solution in place.

SAP API Management has Message Logging Policy to send Syslog messages to third-party log management services. There is a blog already to demonstrate logging to Loggly using it. Message Logging Policy communicates to the log service using TCP or UDP Transport protocol.

In this blog, we will see how to log API interactions to Splunk Cloud using the HTTP Application Protocol. The next blog will focus on using Splunk for SAP CPI, precisely for MPL.

Why HTTP(TLS) and not TCP/UDP?

Splunk for security reasons accepts TCP/UDP connections only from configured forwarders that are installed in the Application that require integration. It is not possible to Install a Splunk Forwarder in SAP API Management (SAP Cloud Platform) and hence we cannot use TCP/UDP Protocol. So the Message Logging Policy is obsolete for Splunk Cloud/Enterprise use case.

Logging Solution

Splunk has a Data Input type called HTTP Event Collector (HEC) which is a fast and efficient way to send data to Splunk Enterprise and Splunk Cloud over HTTP (or HTTPS) Application Protocol. We will use a Service Callout from SAP API Management to log the Events to Splunk Cloud.

About the HTTP Call to Splunk

• The Target URL should be the HTTP Event Collector API
• The Authorization Header should contain Splunk <HEC_Token>
• The Payload should be in the below format when using the JSON Source type.

  {
  	"time": 1589134706278, // timestamp in Unix/epoch format
  	"index": "sapapim_dev", // Index Created in Splunk
  	"host": "dev.apim.com", // A host to identify the event
  	"sourcetype": "_json", // Use JSON standard Source Type
  	"source": "apim.logs", // A source type value of identification
  	"event": {
  		"your_Event_Name": "your_Event_Value"
  	}
  }​

Implementation

Below are the high-level steps to implement Splunk Logging from SAP API Management.

1. Create an Index in Splunk for SAP APIM
2. Create an HEC and associate the Index. This yields an HEC Token.
3. Enable the HEC Token
4. Implement Service Callout in SAP APIM for logging Events.

1 Splunk – Create Index

Log in to Splunk Cloud and open Settings –> Data –> Indexes

Create an Index of your choice. Eg:sapapim_dev

2 Splunk – Create HEC

Open Settings –> Data –> Data Inputs and +Add New  HTTP Event Collector

Enter a Name for the Event Collector

Complete the Input Setting by selecting Source type as _json and App.
Associate the Index created for SAP APIM with this HEC.
Review the Configuration and Submit.
It will yield an HEC Token.

3 Enable HEC Token

Open Settings –> Data Inputs –> HTTP Event Collector –>Global Settings and enable All Tokens.
Enable the token for the HEC just created if not Enabled already.

4 SAP APIM – Service Callout

Open the API Proxy that you need Splunk Logging to be implemented. Add a Service Callout policy in the flow as per your requirement.
In my case I want the request and response payload to be logged to Splunk and hence I have added the Service Callout Policy to Proxy Endpoint –> PostFlow –> Outgoing Response.
Note: Cannot be added to Post Client Flow as it allows only Message Logging policy.
Below is the Policy Code Snippet.

<ServiceCallout async="true" continueOnError="true" enabled="true" xmlns="http://www.sap.com/apimgmt">
	<Request>
		<Set>
			<Headers>
				<Header name="Content-Type">application/json</Header>
				<Header name="Authorization">Splunk e43b4b64-xxxx-xxxx-xxxx-xxxxxxx</Header>
			</Headers>
			<Payload contentType="application/json" variablePrefix="#"
        variableSuffix="@">{
	"time": #client.received.start.timestamp@,
	"index": "sapapim_dev",
	"host": "dev.apim.com",
	"sourcetype": "_json",
	"source": "apim.logs",
	"event": {
		"APIProxy": "Get_OAuthToken",
		"requestVerb": "#request.verb@",
		"ClientStatus": #message.status.code@,
		"reqMessage":"#request.content@",
		"resMessage":#response.content@
	}
}</Payload> 
			<Verb>POST</Verb>
		</Set>
	</Request>
	<!--Remove Response Element if you want Service Callout work Asynchronous--> 
	<Response>splunk.response</Response>
	<Timeout>30000</Timeout>
	<HTTPTargetConnection>
		<URL>https://input-instance.cloud.splunk.com:8088/services/collector/event</URL>
	</HTTPTargetConnection>
</ServiceCallout>

When this policy is executed in the Outgoing Response Flow or Fault Rule or wherever you have added to, a JSON payload in the Splunk required format with API interaction details are logged and indexed in Splunk.

SAP API Management – DIY

• Read my earlier blog SAP API Management – Handling Faults using FaultRules and DefaultFaultRule to know how to add the policy to the Default Fault rule. Use this approach to attach the Service Callout Policy to log the Errors to Splunk.
• Read my blog SAP API Management – API Proxy Troubleshooting Technique or SAP Help Variable Reference to know the runtime variables that can be used for logging.
• You can also create this Splunk logging as a Shared Flow as shown in this blog SAP APIM : Shared Flow and call from other API Proxy Asynchronously by removing the <Response> element from the Service Callout policy.

Result

• • The events can be used to draw Reports and Dashboards
    
  • The events i.e. API Interactions that are logged in Splunk and can be searched for troubleshooting or incident analysis.
    
  • The events can be configured to Send Alert Email/ Create Jira Ticket / Send to Webhook etc in realtime or periodically.

Once you have the data, the result is purely dependent on the capability of the platform and your creativity in utilizing it 🙂