Skip to Content
Technical Articles
Author's profile photo Vandana Gupta

Developing COVID-19 HR assistant Chatbot using SAP Conversational AI including Image processing

Hello Everyone,

Welcome to my first blog post. I hope you all are doing great and are perfectly fit and fine!



As being new to SAP Conversational AI, I have been trying different POCs and integration scenarios with SAP Conversational AI. One of which is integration of Conversational AI Bot with WhatsApp. So here I will cover basic ChatBot development using SAP Conversational AI tools and integrate it with WhatsApp using Twilio. And will use image recognition API for extracting image data. So, Let’s get started!



I understand the COVID-19 pandemic has forced people around the world to adjust to new routines, cope with loneliness, job loss, grief and more. And it’s tough for all of us in different ways, not just physically but mentally. Our government is trying their best to deal with situation is every aspect. But still they can’t reach to every person and have all the details! So keeping this in mind the organisation can help in small manner by providing employees and their family basic details on this.

The Bot will have general conversation with the employee, First BOT will ask first for the employee consent and then their email Id, Afterwards BOT will ask questions like…

  • Do they have any of the following symptoms: fever/feverish, new or existing cough and difficulty breathing?
  • Have they traveled internationally within the last 14 days (outside India)?
  • Have they had close contact with a confirmed or probable COVID-19 case?
  • Have they had close contact with a person with acute respiratory illness who has been outside India in the last 14 days?
  • Are you currently under Quarantine or have been issued a Stay-Home Notice?
  • Then will ask for current living location.
  • Then will ask for employee current temperature reading image.



You can refer to First Bot with SAP Conversational AI for getting started with SAP Conversational AI tools and create a new BOT.

After creation of the BOT,


Step 1: Adding Intents:

 While creating the new BOT I added two predefined skill, Greetings and Small-Talk which will add few related intents automatically. So here I have added only one intent as “YESNO” for the employee reply. It looks something like this.

Here you can see I have added expression “1” for “YES” and “2” for “NO”. It’s for people convenience to reply, We could have used Quick replies for that but as we’re using WhatsApp, it doesn’t support certain format of replies! So we will be doing this way.


Step 2: Building Skills for BOT:

So, my Build looks something like this


So, the conversation will start with user greeting Hi, Hello anything to the BOT. The this will trigger the greetings intents. And in greetings skills actions look something like this



Then if user respond with 1 i.e. ‘YES’ the it will go to email_id skill.



As you can see after the message it will first wait for user input and then redirect to symptoms!

Here in symptoms I have added a requirement which is email which user will provide and it will stored in BOT memory which will be used later to store in database.


Here #email is a gold entity which identifies all the inputs of email type. Similarly there are many such gold entity. We will be using #location and #temperature later to get user location and body temperature and store it.


Once the requirement is fulfilled it will perform the action defined.


In the similar manner we will design our further conversation.


Travel History details:






Close contacted details:



Close contact with other person having any symptoms or recent travel history:



Quarantine Details:



In location skill, the bot will ask the user to enter their current location:



In temperature skill, location is added as a requirement which will be stored in chatbot memory.



Once the requirement is fulfilled, It will ask user for the clear image for their body temperature.



In webhookcall skill, url is added as a requirement which will be stored in chatbot memory as we’re using twilio it will send a media url of the image uploaded.



Step 3: Calling a Webhook

For this we need to connect external services, then call a webhook.


Here is my webhook configuration, I have added a custom body for the payload.

So here the conversation for the chatbot from SAP CAI side is ready!


After this step we will process the image, extract data and then will store the whole data in the database to keep the track.

For image processing I used Imagga . Here you can find amazing collection of image recognition API.

For the middleware purpose of image processing I have created a node.js application and deployed it to the SCP in cloud foundry environment. You can follow this tutorial for the same

Here is my node.js code:

const express = require('express');
const bodyParser = require('body-parser');
const axios = require('axios');
const request = require('request');

const app = express();
app.use(bodyParser.urlencoded({ extended: false }));

//webhook called using CAI tools'/posttest', (req, res) => {
    let config = {
     headers: {'Authorization': "Basic <add you credentials>"}, 
     params: { image_url: req.body.empTemperature},

//making get call to process the image
    axios.get('', config)
    .then(response => {

//creating payload of data to be stored in the database
    var myJSONObject = {                                
        "empEmail" : req.body.empEmail,
        "recentlyTravelled" : req.body.recentlyTravelled,
        "covidSymptoms" : req.body.covidSymptoms,
        "closeContact": req.body.closeContact,
        "maybeCloseContact" : req.body.maybeCloseContact,
        "empTemperature" :[0].data,
        "quarantine" : req.body.quarantine,
        "empLocation" : req.body.empLocation

//Finally making a post to backend and storing the user data to database
        url: <add your post URL>,
        method: "POST",
        json: true,  
        body: myJSONObject
    }, function (error, response, body){
//this is the final response for the webhook called in the format which CAI bot will undersatnd.
        replies: [
                "type": "text",
                "dealy": 2,
                "content": "Here is the value of the temperature: "[0].data 
                "type": "text",
                "content": "Thank you for your efforts to contain the spread. We request you to continue to monitor your health, and watch out for additional announcements from SAP and Government. Your effort is much appreciated. We will contact you next time! Stay safe?"
        conversation: {
          memory: {
            'name': 'PE',
  .catch(error => {
const port = process.env.PORT || 3000;
app.listen(port, function () {
  console.log('myapp listening on port ' + port);


I have created a sample spring boot project for storing the data and have used SAP HANA as database and deployed it to the SAP CLOUD PLATFORM (used SAP Cloud Foundary environment ). It will give you the end-point URL to make the API calls. You can find the code here.



Next step is integrating the bot with WhatsApp.

For that you can follow wonderful SAP CAI-WhatsAPP integration blog post written by Sudip Ghosh.


Here I have add my twilio function code:

exports.handler = function(context, event, callback) {
global.twiml = new Twilio.twiml.MessagingResponse();
var sapcai = require('sapcai');
//when user sends media as response
if(event.MediaUrl0) {
     twiml.message("Please wait! Image is being processed.....");
    console.log("body", event.MediaUrl0);
	let build = new'<your CAI AUTH TOKEN>', 'en')
  build.dialog({ type: 'text', content: event.MediaUrl0}, { conversationId: event.From })
  .then(function(res) {
          var data1 = res.messages[0].content;
          var data2 = res.messages[1].content;
          callback(null, twiml);
//when user sends text as response
} else {
    console.log("body", event.Body);
	let build = new'<your CAI AUTH TOKEN>', 'en')
  build.dialog({ type: 'text', content: event.Body}, { conversationId: event.From })
  .then(function(res) {
        console.log("Response", res.messages);
         var data = res.messages[0].content;
         callback(null, twiml);



Here I have attached a recorded WhatsApp conversation gif of this Integration testing, hope everyone would like it.



I hope you liked the Blog post! You can ask any doubt (If you have any) below in the comment section.

Keep washing your hands and maintain social distancing! Stay safe.







Assigned Tags

      You must be Logged on to comment or reply to a post.
      Author's profile photo Paul PINARD
      Paul PINARD

      Great post Vandana Gupta! Thanks for your interest in SAP Conversational AI!

      Author's profile photo Vandana Gupta
      Vandana Gupta
      Blog Post Author

      Thank you! ?

      Author's profile photo Sudip Ghosh
      Sudip Ghosh

      Great post Vandana Gupta and thank you so much for mention

      Author's profile photo Vandana Gupta
      Vandana Gupta
      Blog Post Author

      Thanks Sudip Ghosh ! Glad you liked it. ?

      Author's profile photo Virendra Sharma
      Virendra Sharma

      Thank you. You have used the Node.js application here. How can we do it with python application?

      Author's profile photo Vandana Gupta
      Vandana Gupta
      Blog Post Author

      Hi Virendra,

      I don't have Python knowledge. But I think you just need to follow the same flow in Python and it will work.

      Author's profile photo Faizullah Mohammed Shaik
      Faizullah Mohammed Shaik

      Hi Vandana, nice blog, where you have demoed how CAI can be helpful during this crisis. thanks for your post.

      Author's profile photo Vandana Gupta
      Vandana Gupta
      Blog Post Author

      Thanks Faiz! Glad you like it ?

      Author's profile photo kratika varshney
      kratika varshney

      Hi Vandana Gupta ,

      Thanks for the nice blog.

      I have to ask one thing, what will be the action and triggers for reply 1 (yes) in the email_id skill. Can you help in that.


      Author's profile photo Subhadip Mandal
      Subhadip Mandal

      Hi Vandana Gupta,

      That's really nice blog with current insights.

      I think this can be integrated with other third-party applications as well, which are having image upload option, of course.

      But the github link you shared for the sample spring boot project, is not working.

      Could you please check and update that please?


      Thank you.

      Author's profile photo Ashwin Katkar
      Ashwin Katkar

      Hi Vandana,

      Hope you are doing well!!

      I was working on integrating SAP CAI with whatsapp with help of twilio and SAPCAI SDK. In Feb 2021 SAP team made mandatory to include OAuth token with developer token which i dont see supported in current SAPCAI SDK on npm.

      Any idea how we can resolve this issue? Here is sample of code for Twilio funtion.

      exports.handler = function(context, event, callback) {
      console.log('invoked with', event)
      global.twiml = new Twilio.twiml.MessagingResponse();
          var sapcai = require('sapcai')
      	let build = new'<CAI APi KEY>', 'en')
        build.dialog({ type: 'text', content: event.Body}, { conversationId: event.From })
        .then(function(res) {
          dta = res.messages[0].content;
          callback(null, twiml);
      Author's profile photo eyemantraa org
      eyemantraa org

      very nice blog

      also view -Eye Mantra Foundation

      Cataract Surgery | Cataract Treatment | Best Cataract Doctors