Introduction

Do you want to build your own smart voice controlled speaker (like Google Home or Amazon Echo) that is responding to natural language input? I do.

Some may ask why bother building it by your own when you can easily buy such devices from anywhere. I think it’s like building a Lego, piece by piece. It’s more fun than buying a complete block, isn’t it?

If you search on the internet, you will see that many people are using it to control other smart-home devices. But I’m thinking what if we use it for business use, instead of personal use? It’s nice to interact with a system through your voice commands, isn’t it?

What I try to achieve here is to build a simple who’s who application using a Raspberry Pi communicating to SuccessFactors. I call it, Project AIM (Assistant in Microdevice).

Table of Contents

• Assumption
• Process flow
• See it in action
• Setup SuccessFactors API
• HCI Configuration
• Dialogflow Configuration
• Setup Raspberry Pi

Assumption

I assume that you must have basic knowledge on SAP Cloud Platform Integration (formerly called SAP HANA Cloud Integration—HCI) and SuccessFactors integration.

Don’t know about Raspberry Pi development? Don’t worry, I have zero knowledge about it when I start this project. Believe me, you will be surprised on how easy it is to set up and find information on the internet. You can also get an AIY Voice Kit from Google—Raspberry Pi + Voice HAT + Raspbian OS with Google Assistant SDK preinstalled.

In this blog, I’m trying not to focus on Raspberry Pi because I think it’s beyond the scope of interests for this community.

Process Flow

See it in action

Setup SuccessFactors API

In this project, I use the SFAPI’s User object so make sure that the adequate authorization has been granted to a SF API user.

HCI Configuration

Step 1: Groovy script – Add a root node in JSON

Why we have to add a root node in the incoming JSON data? Because, as of blog written date, HCI doesn’t support multiple root members processing for the JSON to XML converter. It must contain exactly one member.

So we overcome this limitation using a Groovy script to add a root member to the JSON data.

Script:

import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;
import groovy.json.*

def Message processData(Message message) {
    //Body 
    def jsonOP = message.getBody(String.class);
    jsonOP=jsonOP.toString()
    
    def json_to_str=jsonOP.substring(1, jsonOP.length()- 1);
    json_to_str="{\"Root\": {"+json_to_str+"}}"
    
    message.setBody(json_to_str);
    return message;
}

Dialogflow’s JSON data:

{
  "responseId": "241d8a6d-1df0-4560-91d8-efaba80081f6",
  "queryResult": {
    "queryText": "who is john",
    "action": "GetEmployeeInfo",
    "parameters": {
      "given-name": "John"
    },
    "allRequiredParamsPresent": true,
    "fulfillmentText": "",
    "fulfillmentMessages": [{
      "platform": "ACTIONS_ON_GOOGLE",
      "simpleResponses": {
        "simpleResponses": [{
          "textToSpeech": ""
        }]
      }
    }, {
      "text": {
        "text": [""]
      }
    }]
  }
}

After script processing:

{"Root": 
	{
	  "responseId": "241d8a6d-1df0-4560-91d8-efaba80081f6",
	  "queryResult": {
			...
			...
	  },
	  "originalDetectIntentRequest": {
		"payload": {
		}
	  }
	}
}

Step 2: Groovy script – Log incoming JSON data

If you have experience in SAP PI, I am certain that you will notice one major difference between SAP PI and HCI is tracing capability. It’s becoming a big issue (and frustrated) when you are developing an integration process and trying to find the cause of problems.

As a result, we have to write a script to see processing data in the message processing log.

import com.sap.gateway.ip.core.customdev.util.Message;
import java.util.HashMap;

def Message processData(Message message) {

	def messageLog = messageLogFactory.getMessageLog(message);

	def body = message.getBody(String.class);

	messageLog.addAttachmentAsString("Request",body,"text/xml");	
	message.setProperty("Request", body);

	return message;
}

Step 3: JSON to XML Converter

To manipulate inbound message in HCI including mapping, we have to convert JSON data to XML data first.

Step 4: Message mapping – Dialogflow to SFAPI

We are going to query data from SuccessFactors using SFAPI based on incoming input from Dialogflow. So the parameter given-name will be mapped with the field data_output.

Step 5: Content Modifier – Take “Name” from XML

The parameter “EmpName” will be passed to SFAPI in order to query data from its User object. The value is taken from the XML path /data_output (from prior step).

Step 6: Request-Reply – Get data from SuccessFactors

Retrieve data from SuccessFactors using SFAPI, the parameter “EmpName” ($(header.EmpName)) is put in the where condition to select a specific person.

Step 7: Message mapping– SFAPI to Output

In this step, the output data from SuccessFactors will be mapped to the output field fulfillmentText. You can also use a Groovy script to map multiple fields, instead of using graphical mapping. Choice is yours.

Step 8: XML to JSON Converter

This step is basically to convert processed XML data to JSON data and send back to Dialogflow for further processing.

Step 9: Groovy script – Log outgoing JSON data

Similar to the step 2, but this time is intended for logging outbound data.

Step 10: Content Modifier – Set the content type

The Dialogflow’s webhook is required with the header of content type to be application/json. So we use the content modifier to set the header as well as set the message body with data from the prior step.

Last step: Outbound data

Once the end message is reached, the processed data will automatically be sent back to the sender (Dialogflow).

Dialogflow Configuration

What is Dialogflow? It’s a platform powered by Google’s machine learning for building conversational interfaces for IoT devices, mobile applications, and other popular platforms.

How it’s working? A Dialogflow agent recognizes the intent (representing a mapping between what you say and what action will be taken) and context of what you say. The agent is triggered by an invocation to request and respond your question.

You can find more information on https://www.dialogfow.com.

Create a new agent

Go to https://console.dialogflow.com/api-client and select Create a new agent.

Put a meaningful agent name. Then click CREATE.

Create an intent

Put an intent name and click SAVE. After that click on ADD TRAINING PHRASES under the Training phrases section.

This is to define possible phrases that you are going to say to trigger the intent. The more you define, the easier for you that the intent will be triggered. When the phrases are defined, Dialogflow automatically highlights those keywords (by its machine learning engine) called entity. The entity and resolved value are then used within the request.

Now we are going to define the action corresponds to your request. Click MANAGE PARAMETERS AND ACTION under the Action and parameters section.

Define the action name GetEmployeeInfo as well as the parameter name linking with the entity.

Define default responses when you make a request. Click ADD RESPONSE under the Responses section.

Put any text responses as you like.

Now, at the fulfilment step, Dialogflow has adequate information to pass through the request to fulfil your request. Technically data will be sent to what it’s called webhook (web service that receives a HTTP POST request). Once the webhook processed your request successfully, information will be sent back to Dialogflow.

To create a fulfilment, you must enable it first by clicking ENABLE FULFILMENT under the Fulfilment section. And then turn on Enable webhook call for this intent.

Create a Fulfilment

There are several choices of development for you to do it. Many people are using Node.JS or its inline editor (that is powered by Google’s Cloud Functions for Firebase). For me, I have experience in HCI, so I decide to use this one.

If everything is set, you can test it out on the right-hand side.

Click on SHOW JSON, you can also see the outbound JSON data from Dialogflow to your web service.

At this point, you have done almost everything you have to do in Dialogflow. Remember that Dialogflow will be triggered by Actions for Google Assistant?

Now we are going to link it together. Click Integrations on the left-hand side menu, then click INTEGRATION SETTINGS in the Google Assistant section.

Click MANAGE ASSISTANT APP.

Click Add, and fill in the details of your application as well as other required fields.

If everything is set, you can also test the application in the Simulator.

Now we will enable Google API on Google Cloud Platform (GCP). Go back to Dialogflow and click on the project ID under the Google project section. You will be redirected to GCP.

Turn on the Google Assistant API in API Library.

To access Google API, it doesn’t use the user/password authentication. Instead it uses OAuth 2.0 authentication.

In the Cloud Console, create an OAuth 2.0 client by going to APIs & Services > Credentials.

Create a new client ID with the Other type.

A dialog window will pop up. Click OK. In the Credentials list, find your new credentials and click the download icon on the right-hand side. The file is in the JSON format.

Copy the JSON file (client_secrets_XXXX.json) to your Raspberry pi device in the path /home/pi/. Then rename it to assistant.json.

Setup Raspberry Pi

If you use Google AIY Voice Kit, the setup is pretty easy. You can follow the steps in https://aiyprojects.withgoogle.com/voice. It comes with a Raspberry Pi, Voice HAT, and customized Raspbian OS with Google Assistant SDK preinstalled.

Or if you have only a Raspberry Pi, then you need to buy a Voice HAT and set it up. As well, you need to install Google Assistant SDK to make a call from Raspberry Pi to Dialogflow via Google Assistant API.

Write the codes (Python) in Raspberry Pi

Actually there are many programming languages (Python, C, C++, Java, etc.) you can use to develop programs for Raspberry Pi. Why Python? Because when I start this project, I know absolutely nothing about Raspberry Pi programming and Python is quite easy to learn and use. Also since it’s one of very popular programming language, so you can easily get help from anywhere on the internet.

To develop a Python program, there are various choices of IDE you can pick. Since I’m not intending to develop such a complex program, so I use Thonny IDE. It has a clean user interface and very easy to use.

The Python program to initiate a request

The following codes are excerpted from Google AIY Voice project in Github. You can initiate the request by both pressing the button and saying the keyword.

import platform
import sys
import threading

import aiy.assistant.auth_helpers
from aiy.assistant.library import Assistant
import aiy.voicehat
from google.assistant.library.event import EventType

class MyAssistant(object):

    def __init__(self):
        self._task = threading.Thread(target=self._run_task)
        self._can_start_conversation = False
        self._assistant = None

    def start(self):
        self._task.start()

    def _run_task(self):
        credentials = aiy.assistant.auth_helpers.get_assistant_credentials()
        with Assistant(credentials) as assistant:
            self._assistant = assistant
            for event in assistant.start():
                self._process_event(event)

    def _process_event(self, event):
        status_ui = aiy.voicehat.get_status_ui()
        if event.type == EventType.ON_START_FINISHED:
            status_ui.status('ready')
            self._can_start_conversation = True
            # Start the voicehat button trigger.
            aiy.voicehat.get_button().on_press(self._on_button_pressed)
            if sys.stdout.isatty():
                print('Say "OK, Google" or press the button, then speak. '
                      'Press Ctrl+C to quit...')

    def _on_button_pressed(self):
        if self._can_start_conversation:
            self._assistant.start_conversation()

def main():
    MyAssistant().start()

if __name__ == '__main__':
	main()

Finished!