Skip to Content
Technical Articles

Developing COVID-19 HR assistant Chatbot using SAP Conversational AI including Image processing

Hello Everyone,

Welcome to my first blog post. I hope you all are doing great and are perfectly fit and fine!

 

BACKGROUND 

As being new to SAP Conversational AI, I have been trying different POCs and integration scenarios with SAP Conversational AI. One of which is integration of Conversational AI Bot with WhatsApp. So here I will cover basic ChatBot development using SAP Conversational AI tools and integrate it with WhatsApp using Twilio. And will use image recognition API for extracting image data. So, Let’s get started!

 

Overview

I understand the COVID-19 pandemic has forced people around the world to adjust to new routines, cope with loneliness, job loss, grief and more. And it’s tough for all of us in different ways, not just physically but mentally. Our government is trying their best to deal with situation is every aspect. But still they can’t reach to every person and have all the details! So keeping this in mind the organisation can help in small manner by providing employees and their family basic details on this.

The Bot will have general conversation with the employee, First BOT will ask first for the employee consent and then their email Id, Afterwards BOT will ask questions like…

  • Do they have any of the following symptoms: fever/feverish, new or existing cough and difficulty breathing?
  • Have they traveled internationally within the last 14 days (outside India)?
  • Have they had close contact with a confirmed or probable COVID-19 case?
  • Have they had close contact with a person with acute respiratory illness who has been outside India in the last 14 days?
  • Are you currently under Quarantine or have been issued a Stay-Home Notice?
  • Then will ask for current living location.
  • Then will ask for employee current temperature reading image.

 

CREATING THE BOT

You can refer to First Bot with SAP Conversational AI for getting started with SAP Conversational AI tools and create a new BOT.

After creation of the BOT,

 

Step 1: Adding Intents:

 While creating the new BOT I added two predefined skill, Greetings and Small-Talk which will add few related intents automatically. So here I have added only one intent as “YESNO” for the employee reply. It looks something like this.

Here you can see I have added expression “1” for “YES” and “2” for “NO”. It’s for people convenience to reply, We could have used Quick replies for that but as we’re using WhatsApp, it doesn’t support certain format of replies! So we will be doing this way.

 

Step 2: Building Skills for BOT:

So, my Build looks something like this

 

So, the conversation will start with user greeting Hi, Hello anything to the BOT. The this will trigger the greetings intents. And in greetings skills actions look something like this

 

 

Then if user respond with 1 i.e. ‘YES’ the it will go to email_id skill.

 

 

As you can see after the message it will first wait for user input and then redirect to symptoms!

Here in symptoms I have added a requirement which is email which user will provide and it will stored in BOT memory which will be used later to store in database.

 

Here #email is a gold entity which identifies all the inputs of email type. Similarly there are many such gold entity. We will be using #location and #temperature later to get user location and body temperature and store it.

 

Once the requirement is fulfilled it will perform the action defined.

 

In the similar manner we will design our further conversation.

 

Travel History details:

 

 

 

 

 

Close contacted details:

 

 

Close contact with other person having any symptoms or recent travel history:

 

 

Quarantine Details:

 

 

In location skill, the bot will ask the user to enter their current location:

 

 

In temperature skill, location is added as a requirement which will be stored in chatbot memory.

 

 

Once the requirement is fulfilled, It will ask user for the clear image for their body temperature.

 

 

In webhookcall skill, url is added as a requirement which will be stored in chatbot memory as we’re using twilio it will send a media url of the image uploaded.

 

 

Step 3: Calling a Webhook

For this we need to connect external services, then call a webhook.

 

Here is my webhook configuration, I have added a custom body for the payload.

So here the conversation for the chatbot from SAP CAI side is ready!

 

After this step we will process the image, extract data and then will store the whole data in the database to keep the track.

For image processing I used Imagga . Here you can find amazing collection of image recognition API.

For the middleware purpose of image processing I have created a node.js application and deployed it to the SCP in cloud foundry environment. You can follow this tutorial for the same

Here is my node.js code:

const express = require('express');
const bodyParser = require('body-parser');
const axios = require('axios');
const request = require('request');

const app = express();
app.use(bodyParser.urlencoded({ extended: false }));
app.use(bodyParser.json());

//webhook called using CAI tools
app.post('/posttest', (req, res) => {
    let config = {
     headers: {'Authorization': "Basic <add you credentials>"}, 
     params: { image_url: req.body.empTemperature},
    }

//making get call to process the image
    axios.get('https://beta-api.imagga.com/v2/text', config)
    .then(response => {

//creating payload of data to be stored in the database
    var myJSONObject = {                                
        "empEmail" : req.body.empEmail,
        "recentlyTravelled" : req.body.recentlyTravelled,
        "covidSymptoms" : req.body.covidSymptoms,
        "closeContact": req.body.closeContact,
        "maybeCloseContact" : req.body.maybeCloseContact,
        "empTemperature" : response.data.result.text[0].data,
        "quarantine" : req.body.quarantine,
        "empLocation" : req.body.empLocation

    };
//Finally making a post to backend and storing the user data to database
    request({
        url: <add your post URL>,
        method: "POST",
        json: true,  
        body: myJSONObject
    }, function (error, response, body){
        console.log(response.body);
    });
//this is the final response for the webhook called in the format which CAI bot will undersatnd.
    res.json({
        replies: [
            {
                "type": "text",
                "dealy": 2,
                "content": "Here is the value of the temperature: " +response.data.result.text[0].data 
            },
            {
                "type": "text",
                "content": "Thank you for your efforts to contain the spread. We request you to continue to monitor your health, and watch out for additional announcements from SAP and Government. Your effort is much appreciated. We will contact you next time! Stay safe😊"
            }
            
          ],
        conversation: {
          memory: {
            'name': 'PE',
          }
        }
    });
  })
  .catch(error => {
    console.log('error',error);
  });
});
const port = process.env.PORT || 3000;
app.listen(port, function () {
  console.log('myapp listening on port ' + port);
});

 

I have created a sample spring boot project for storing the data and have used SAP HANA as database and deployed it to the SAP CLOUD PLATFORM (used SAP Cloud Foundary environment ). It will give you the end-point URL to make the API calls. You can find the code here.

 

 

Next step is integrating the bot with WhatsApp.

For that you can follow wonderful SAP CAI-WhatsAPP integration blog post written by Sudip Ghosh.

 

Here I have add my twilio function code:

exports.handler = function(context, event, callback) {
global.twiml = new Twilio.twiml.MessagingResponse();
var sapcai = require('sapcai');
//when user sends media as response
if(event.MediaUrl0) {
     twiml.message("Please wait! Image is being processed.....");
    console.log("body", event.MediaUrl0);
	let build = new sapcai.build('<your CAI AUTH TOKEN>', 'en')
  build.dialog({ type: 'text', content: event.MediaUrl0}, { conversationId: event.From })
  .then(function(res) {
      'debugger';
          var data1 = res.messages[0].content;
          var data2 = res.messages[1].content;
          twiml.message(data1);
          twiml.message(data2);
          callback(null, twiml);
  },3000);
//when user sends text as response
} else {
    console.log("body", event.Body);
	let build = new sapcai.build('<your CAI AUTH TOKEN>', 'en')
  build.dialog({ type: 'text', content: event.Body}, { conversationId: event.From })
  .then(function(res) {
      'debugger';
        console.log("Response", res.messages);
         var data = res.messages[0].content;
         twiml.message(data);
         callback(null, twiml);

  })	
}
};

 

Here I have attached a recorded WhatsApp conversation gif of this Integration testing, hope everyone would like it.

 

 

I hope you liked the Blog post! You can ask any doubt (If you have any) below in the comment section.

Keep washing your hands and maintain social distancing! Stay safe.

 

 

 

 

 

 

9 Comments
You must be Logged on to comment or reply to a post.