Technical Articles

Ferry Djaja

January 30, 2019 8 minute read

Facebook Messenger OCR Receipt Bot

I would like to share how to create a Facebook Messenger bot that has a capability to perform OCR (Optical Character Recognition) on the image of a receipt taken by a mobile phone via Facebook Messenger platform. User need to upload the receipt to FB messenger and the bot will reply with the OCR result.

We will be using the OCR engine with the following options:

SAP Leonardo Machine Learning Foundation (cloud based)
Our own OCR engine with Tesseract OCR (installed on local machine)

User can choose either to use SAP Leonardo or Tesseract solution based from the Facebook Messenger bot menu interface.

Architecture Diagram

The architecture contains two main components

ScanReceiptBotApp is a NodeJS app running on SAP Cloud Foundry which handles the bot conversation status (getting and processing the receipt image) and performing the back end calls to Tesseract OCR engine and web calls to SAP Leonardo Inference Service for OCR.
Tesseract OCR Engine is a Python flask app that runs on-premise (local machine) to convert the image receipt to OCR and send the result back to ScanReceiptBotApp.

The connectivity between back end system and SAP Cloud Foundry is bridged using SAP Cloud Connector.

Pre-requisite Components

We need to have the account for SAP Cloud Foundry and Facebook Developer.

SAP Cloud Foundry subaccount
SAP API Business
Facebook developer account to create the Facebook Messenger Bot

Install the following software on your local machine (I am using Ubuntu 18.04.1 LTS)

SAP Cloud Connector (2.11.3)
Python (3.6.7)
Tesseract (4.0.0-beta.1)
OpenCV (4.0.0)
NodeJS (8.11.4)

Let’s go through the subsequent steps to build one!

Step 1 – Configure SAP Leonardo OCR

Go to SAP API Business Hub https://api.sap.com/ and register an account.
Search for “SAP Leonardo Machine Learning Foundation – Functional Services” and click it to go to next step.

On the next screen, search for “ocr” and select the “Inference Service for Optical Character Recognition“.

Click on Show API Key and take note the key. We will use the key later in the Facebook Messenger Bot app.

Step 2 – Configure Tesseract OCR Engine

We will create a Python flask server app which read the image from the URL and convert it to text.

To improve the quality of the OCR result, the input image is first pre-processed to yield a scanner effects as follow:

We will not cover all the details for each box, you may refer to this great article if you are interested to find out more: https://www.pyimagesearch.com/2018/09/17/opencv-ocr-and-text-recognition-with-tesseract/

Install the required libraries and run the server with this command. The app is running on port 3000.

python OCRserver.py

To perform the OCR on the image from the URL (example: https://d85ecz8votkqa.cloudfront.net/support/help_center/Register_Order_Receipt.JPG), execute the HTTP Get request from Postman and we will get the JSON response back if there is no error:

http://localhost:3000/scan?url=https://d85ecz8votkqa.cloudfront.net/support/help_center/Register_Order_Receipt.JPG

The result is not too bad!

Step 3 – Configure SAP Cloud Foundry

We will create two apps that will be hosted in SAP Cloud Foundry:

appnode-demo which handles the connectivity to OCR Tesseract Engine. We will be using SAP Cloud Connector for this purpose since.
ScanReceiptBotApp which is NodeJS Facebook Bot app.

appnode-demo

Configure the SAP Cloud Connector for OCR Tesseract Engine that is hosted locally.
You can see the OCR Tesseract Engine is running on my local machine with port 3000.

On SAP Cloud Foundry, set the destination pointing to my local machine.

Create the connectivity, destination and xsuaa service instances. You may refer to my blog how to do it.

Deploy the app. Please find the config files on my Git.
Take note the app routes URL (https://appnode-demo-pxxxxxtrial.cfapps.eu10.hana.ondemand.com/). We will use this information on the Facebook Bot app.

ScanReceiptBotApp

Create a folder in your computer and name it as bot1 and create the following files: app.js and manifest.yml. Full source code is here.
On app.js, we need to update the following lines:
ocr: SAP Leonardo OCR token we got it from “Step 1 – Configure SAP Leonardo OCR”.
token: Facebook page token that we will get in the next step.
ocrhost: appnode-demo application routes URL that we got it from “Step 3 – Configure SAP Cloud Foundry”.
uid: is your SAP Cloud Foundry user id.
pwd: is your SAP Cloud Foundry password.

// set the SAP Leonardo OCR API Key
const ocr = new OCR("SAP_LEONARDO_TOKEN");
// set the Facebook Messenger Token
const token = "FACEBOOK_TOKEN"
// Set the Cloud Foundry OCR App URL with base authentication type
const ocrhost = "appnode-demo-pxxxxxtrial.cfapps.eu10.hana.ondemand.com";
const uid = "SAP_CLOUDFOUNDRY_ID";
const pwd = "SAP_CLOUDFOUNDRY_PASSWORD";

Install the following components:

npm install express body-parser request jsonfile fs https assert

Take note the app is running on port 5000. You can run the app locally with this command to test and see if there is any error. We will later deploy the app to SAP Cloud Foundry.

node app.js

This the function that interact with SAP Leonardo and Tesseract OCR. Basically we grab the image from Facebook Messenger attachment, feed it to the OCR function and get the result.

//Processing
if (event.message && event.message.attachments) {
    if (event.message.attachments[0].type === "image") {
        let imageURL = event.message.attachments[0].payload.url;
        console.log(imageURL);

        /*
        //For local testing
        request('http://localhost:3000?url=' + imageURL, { json: true }, (err, res, body) => {
          if (err) { return console.log(err); }
          console.log(body);
          sendTextMessage(sender, body.result.ocr, function(returnValue) {
          })
        });
        */

        if (SAPOCR) {
            //Using SAP Leonardo

            sendTextMessage(sender, "Processing OCR with SAP Leonardo. Please wait...", function(returnValue) {})

            ocr.ocr(imageURL)
                .then((body) => {
                    console.log(body);
                    sendTextMessage(sender, body.predictions[0], function(returnValue) {})
                })
                .catch((err) => {
                    console.error(err);
                });

        } else {
            //Using Tesseract

            sendTextMessage(sender, "Processing OCR with Tesseract. Please wait...", function(returnValue) {})

            let options = {
                host: ocrhost,
                path: '/scan?url=' + imageURL,
                headers: {
                    'Authorization': 'Basic ' + new Buffer(uid + ':' + pwd).toString('base64')
                }
            };

            https.get(options, function(res) {
                let body = "";
                res.on('data', function(data) {
                    body += data;
                });
                res.on('end', function() {
                    //here we have the full response, html or json object				        
                    let ocrresult = JSON.parse(body);
                    console.log(ocrresult);

                    sendTextMessage(sender, ocrresult.result.ocr, function(returnValue) {})
                })
                res.on('error', function(e) {
                    console.log("Got error: " + e.message);
                });
            });
        }

    }
}

Facebook Bot Messenger Setup

Create a Facebook page to host the bot.

Logon to https://developers.facebook.com/ and create a new app ID.
Enter Display Name, Contact Email and click Create App ID.

Select Products and add Messenger. Click Set Up to continue.

On the Messenger Platform page, scroll down to Token Generation section. Select the page you created (for my case is OCRBot).

Authorize the page and click OK to continue.

The page access token will be generated. Copy that token and we will use in ScanReceiptBotApp: app.js.

Edit the ScanReceiptBotApp: app.js we created earlier and assign the token to the token variable on line 77. Save the file.

Deploy the app to SAP Cloud Foundry with command cf push.

Wait until deploy is completed and check if there is any error. Take note the routes URL information.

Go back to Facebook Developer site and navigate to Webhooks section. Click Setup Webhooks.

On New Page Subscription, fill in the following details:
Callback URL: is the URL of ScanReceiptBotApp on SAP Cloud Foundry with suffix /webhook: https://scanreceiptbotapp.cfapps.eu10.hana.ondemand.com/webhook.
Verify Token: my_voice_is_my_password_verify_me
Subscription Fields: messages, messaging_postbacks and message_deliveries.
Click Verify and Safe to continue.

Check if status is Complete.
Select OCRBot page and click Subscribe. In the end you should have similar setup as per below screenshot.

Please note that the bot is in the Development status and is not available for public yet. You need to submit to Facebook and get it approve before public can use your bot. That topic will not be covered on this post.

Testing the Bot

All setups have been completed now, we can try to test the bot.

Make sure the Tessseract OCR engine is running on your local machine.
Install the Facebook Messenger App on your phone.
Search for OCRBot > Click Get Started.
Select the OCR mode option: using SAP Leonardo or Tesseract OCR.

Take a picture of receipt or select the existing receipts from your gallery. Send it to the bot.
Wait for a while before the bot response with the OCR result.

What’s Next ?

I think the OCR result is not too bad and of course a lot of improvements need to be done. I am thinking to apply the NLP on the OCR result so user can get the summary of it. Until next time ! Thanks.

Be the first to leave a comment

You must be Logged on to comment or reply to a post.