Sentiment Analysis using SAP Analytics Cloud + Google Cloud Natural Language Processing API

[eaksiri]

Introduction

SAP Analytics Cloud (SAC) is such a powerful tool for analytics and creates compelling data visualizations. And I intend to write this blog for studying if I can add the sentiment analysis capability through the pre-trained machine learning models from Google Cloud Natural Language to SAC.

Assuming you have an online ticketing system for customer support. After some time, you want to measure the quality of service that you have been providing through customer’s feedback. So we will use the sentiment analysis to determine the overall attitude (either positive or negative) based on the feedback provided.

Ultimately the ticket data will be enriched with the sentiment score to define on how well is your support.

 

Table of Contents

• Prerequisites


• Process Flow


• Enable Cloud Natural Language API on Google Cloud Platform


• Build the integration scenario on SAP Cloud Integration


• Create a dashboard on SAP Analytics Cloud


• Conclusion

 

Prerequisites

• SAP Analytics Cloud account


• SAP Cloud Integration account


• Google Cloud Platform account

 

Process Flow

The dashboard gets data feed from SAP Cloud Integration (SCI) via OData. SCI then gets feedback data from the ticketing system with the REST API. After that, the sentiment score is determined based on individual sentence that a customer puts in the ticket response using Google Cloud Natural Language API.

Finally the processed data will be sent back to SAC to generate the dashboard.

 

Enable Cloud Natural Language API on Google Cloud Platform

Create a new project

Login to Google Cloud Platform Console. Then create a new project by going to Manage Resources Page and click on CREATE PROJECT.



Put a project name and click CREATE. Also make sure that you enable billing for your account.



 

Enable the Cloud Natural Language API

Go to the Dashboard menu and click on APIs & Services. Then click on Enable APIS AND SERVICES.



 

Create an API Key

Go to the Credentials section of APIs & services. Then click Create credentials drop-down and select API Key.



The pop-up window will be shown with the generated API key. Copy the API key and save it.



 

Testing: Call the API to perform Sentiment Analysis

You may have to check if your configuration is correct. I usually use Postman for testing the REST APIs.

We use the following endpoint to send the request along with the API key from the above step to the API.

• https://language.googleapis.com/v1/documents:analyzeSentiment?key=${API_KEY}

The Natural Language API is a REST API consisting of JSON request and response. A simple JSON request is as follows:

{

 "document": {

  "content": "Very good support.",

  "type": "PLAIN_TEXT"

 },

 "encodingType": "UTF8"

}

Document contains the data for this request, which consists of the following fields:

• content – contains the text to evaluate


• type – document type (HTML or PLAIN_TEXT)


• encodingType – character encoding

Let’s try it on Postman.



You can get the sentiment score based on the provided sentence. The Sentiment score ranges from -1 (very bad) to +1 (very good). From the above response, the sentiment score is 0.8 which means quite good.

 

Build the integration scenario on SAP Cloud Integration

(1) OData Sender Adapter

The OData sender adapter is to be triggered from SAP Analytics Cloud with the following entity set structure.

Entity Set	Entity	Description
ContentSet	SurveyID	Survey ID
	Content	Feedback data from the users
	Group	Group of support team
	SentimentScore	Sentiment score ranges from -1 to +1

(2) Process Call: Support Ticket API

This step is to retrieve feedback data from the support ticketing system (in my case, I use Freshdesk as an example). I’m not going to focus the details here because you may use different ticket tools such as Jira or Salesforce. Anyway, most of the ticketing systems offer the data access through REST API. So you can use your own and easily follow through the rest of the blog.

Sample XML data from the Freshdesk API:

   <element>

      <agent_id/>

      <feedback>Very good support</feedback>

      <group_id>Customer Support</group_id>

      <id>44000002882</id>

      <survey_id>44000001495</survey_id>

      <ticket_id>5</ticket_id>

      <updated_at>2018-11-15T06:33:07Z</updated_at>

      <user_id>44000067346</user_id>

   </element>

 

(3) Iterating Splitter

To split multiple feedback data into a single element because the Google Cloud Natural Language Processing (NLP) API can only receive only one input at a time.



 

(4) Content Modifier: Save feedback data for further processing

We will use the split feedback data (from the prior step) later to be combined with the returned sentiment score taken from the Google NLP API. The feedback data is saved in the property called FeedbackData.



 

(5) Message Mapping: Construct a request for sentiment analysis API

The Google Cloud NLP API’ sentiment analysis is using the following JSON request.

"document":{

  "content": <TEXT>,

  "type":"PLAIN_TEXT"

}

So we need to map the incoming feedback data to the request data in the XML format first. Then convert it to the JSON format in the later step.



 

(6) XML To JSON Converter

Convert the XML data (from the prior step) and suppress the JSON root element.



 

(7) Request Reply: HTTP Request

Make a call to the sentiment analysis API.



 

(8) Groovy script: Add a root node in JSON

Since HCI doesn’t support multiple root members processing for the JSON to XML converter. It must contain exactly one member. So we have to add a root node in the JSON data.

import com.sap.gateway.ip.core.customdev.util.Message;

import java.util.HashMap;

import groovy.json.*



def Message processData(Message message) {

    //Body 

    def jsonOP = message.getBody(String.class);

    jsonOP=jsonOP.toString()

    

    def json_to_str=jsonOP.substring(1, jsonOP.length()- 1);

    json_to_str="{\"Root\": {"+json_to_str+"}"

    

    message.setBody(json_to_str);

    return message;

}

 

(9) JSON To XML Converter

To convert the returned sentiment data in the JSON format to the XML format for further processing.

 

(10) Filter: Node documentSentiment

As we focus only the sentiment score, so we’ll filter the rest of the message out.





 

(11) Content Modifier: Combine feedback data with sentiment score

This step is to combine feedback data (from the step 4) and the returned sentiment score (from the step 10).

<root>

  <document>

    <documentSentiment>

      <magnitude>0.9</magnitude>

      <score>0.9</score>

    </documentSentiment>

    <element>

      <agent_id null="true"/>

      <created_at>2018-11-15T06:33:07Z</created_at>

      <feedback>Very good support</feedback>

      <group_id>Customer Support</group_id>

      <id>44000002882</id>

      <ratings>

        <default_question>103</default_question>

      </ratings>

      <survey_id>44000001495</survey_id>

      <ticket_id>5</ticket_id>

      <updated_at>2018-11-15T06:33:07Z</updated_at>

      <user_id>44000067346</user_id>

    </element>

  </document>

</root>

 

(12) Gather: Collect data among the iteration steps

 



 

(13) Filter: Remove Multimap node

To remove the unnecessary Multimap node from the Gather step.





 

(14) Groovy script: Add a root node in JSON

It’s as same as the step 8.

 

(15) Message Mapping: Construct the OData output

Almost there! Map the outcome to the sender’s OData structure.



 

(16) Last Step

Once the end message is reached, the processed data will automatically be sent back to the sender.

Last but not least, you need the OData endpoint to be called from SAP Analytics Cloud. You can get it from Manage Integration Content as highlighted.



 

Create a dashboard on SAP Analytics Cloud

Create a connection

Login to SAP Analytics Cloud. Then open the dashboard menu and select Connection.



 

After that click the plus button and select Import Data Connection --> OData Services.



 

Put the connection name and the data service URL with the OData endpoint from the prior section.



 

Create a story

Open the dashboard menu again, and select Create --> Story.



 

Select the Dashboard template.



 

Now we going to use data from our integration scenario. Select the Data from the top-left panel.



 

Choose Data acquired from a datasource.



 

Pick OData Services as a datasource.



 

Choose the connection that you created from the earlier step.



 

Next, name your query and select the query set (it’s your entity set that you use in the SCI’s OData sender step).



 

You can then select the data you are interested in to build a query.



 

After your query is created, data will be retrieved from your integration scenario via OData and gets previewed in the subsequent screen.



 

Now we are going to build a dashboard. Select Story from the top-left panel. In this blog, I use the predefined layout Table and content. Or you can select another layout as you wish and then click Apply.



 

The layout Table and content is composed of 3 indicators on the top and one table below.



 

I use the table to display the raw data I get from the integration content. There are 3 rows selected—SurveyID, Group, and Content—considering as dimensions. And the SentimentScore is considered as a measurement.



 

The first indicator (on the top-left corner) I’ll use it for presenting the average sentiment score based on multiple supporting tickets.

The indicator I use is Numeric Point. To calculate the score average, select Create Calculation under Primary Values.



 

In Calculation Editor, choose the operation AVERAGE with the measurement SentimentScore.



 

The average sentiment score will then be shown on your dashboard.



 

To make it easier to determine whether your received feedback is positive or negative, we are going to colorize the indicator, says green for positive and red for negative.

Select Add Thresholds under Color. Define the number range between 0 and 1 for the positive feedback and the negative feedback will be ranged between -1 to 0.



 

Click Apply and check out the real-time result.



 

Now I’m going to build a second indicator for checking the feedback from different support groups—e.g. Customer Support, Integration, SAP, and SuccessFactors.

Select the comparison Bar/Column. Then select the measurement Sentiment with the dimension Group. You can also try to colorize the indicator as you like.



 

Here you go, you get the average feedback score as well as the sentiment score for each support group.



 

You can play around with the last indicator as you wish.



 

Conclusion

SAP Analytics Cloud also has a build-in machine learning feature called Smart Discovery. Its feature is to find correlations among your dataset and uncover valuable information. But it cannot identify emotional opinion within the text or sentence to determine where it’s positive or negative.

Perhaps I’m using a trial account so that feature is not available for me. Anyway I think it’s still exciting and fun to play around with other options available in the market, don’t you think?

I hope this blog is helpful in one way or another.