Create Text Recognition (OCR) App with Google ML Kit and SAP Web IDE

In this tutorial, I would like to go through on how to build the text scanner (OCR) app with Google ML and SAP Web IDE. The app that we are going to build is the Android Cordova SAPUI5 that has two functions: Take Photo and Pick from Gallery.

Take Photo

To snap a photo and feed into the text recognizer and get the OCR result.

Pick from Gallery

To feed the text recognizer with the available existing images from the phone gallery and get the OCR result.

Once it get recognized, the dialog box with the OCR result will pop-up. We can then use this information to proceed further like getting the information from the SAP back-end through oData, etc.

The final app will look like this:

Let’s go through the following steps.

Configure Local HAT

Configure Local HAT installation by following this blog: Check my previous blog on how to install SAP HAT on your machine: https://blogs.sap.com/2019/01/07/build-android-sapui5-ocr-scanner-with-sap-web-ide-and-anyline-ocr-sdk/.

Download Plugins

Download all plugins to local custom plugin folder.

• Cordova ML Text Plugin
  https://bitbucket.org/bhivedevs/cordova-plugin-ml-text.git.
• Cordova  Plugin Firebase
  https://www.npmjs.com/package/cordova-plugin-firebase
• Cordova Plugin Camera
  We will configure in Web IDE

Update Local HAT Installation

Since the Cordova ML Text plugin requires cordova 7.1.0+ , cordova android 6.4.0+, we need to modify the local SAP HAT installation to meet this requirement.

On the SAP HAT Local folder, go to ./setup/server/routes/check.js and modify line 302 to use cordova android 6.4.0:

cmd: 'cordova platform add ' + platform + (platform==="android"?"@6.4.0":""),	// for android sdk tools > 26.0.1

Run HAT

Now run HAT setup from command line and make sure status is passed.

Once the setup is completed, run the HAT server from command line.

Configure SAP Web IDE

Open SAP Web IDE Full Stack and go to Preferences > Hybrid Application Toolkit. Click Test Connection and make sure the connection is available.

Create a new project from template.

Go to Project Settings > Hybrid App Toolkit > App Configuration

Provide the app name Scanner and app id com.demo.scanner and select Android since we are going to build the android app.

Under Cordova plugin, select camera.

Under Custom plugin, select Google Firebase and Mltext and click Save.

Deploy the app. Right click on the app that we created and select Deploy > Prepare Hybrid Project.

Preparation in progress. It is also installing the custom plugins.

Modify the Plugins

Once it is done, go to C:\Users\<user>\SAPHybrid folder. The project Scanner is created there.

The created cordova project is still using version 6.2.3 and it will not meet the Cordova ML Text plugin (6.4.0). Let’s update it to 6.4.0.

Go to C:\Users\<user>\SAPHybrid\Scanner\hybrid\platforms\android and run this command:

I checked C:\Users\<user>\SAPHybrid\Scanner\hybrid\platforms\android\res\xml\config.xml and it looks like the Cordova ML Text plugin was not added successfully.

Let’s add it manually then, run this command:

cordova plugin add https://bitbucket.org/bhivedevs/cordova-plugin-ml-text.git

Now we got the plugin installed:

We also need to remove the cordova-plugin-compat otherwise we will have an error during build process. Refer to this https://stackoverflow.com/questions/46562289/multiple-dex-files-define-lorg-apache-cordova-buildhelper

And this would be the final plugin list:

Create the Firebase Config File

Go to Firebase console https://console.firebase.google.com and create a new project.

Go to Project settings and click Add Firebase to your Android App.

Add android package name com.demo.scanner and click Register app.

Download google-services.json.

And place it under C:\Users\<user>\SAPHybrid\Scanner\hybrid\platforms\android

Import to Android Studio

Open Android Studio and import the project C:\Users\<user>\SAPHybrid\Scanner\hybrid\platforms\android

Update the build.gradle: Android and CordovaLib project

Update build.gradle: Android:

Update build.gradle: CordovaLib:

And now you can build the app without error.

Main Program

Take a look at the function below from the controller from the asset folder.

mltext.getText was called to recognize the text in the image and will get the result from recognizedText.

return Controller.extend("GPS.GPS.controller.GPS", {

    oView: null,

    onInit: function() {
        oView = this.getView();
    },

    onPhotoDataSuccess: function(imageData) {
        var myImage = oView.byId("myImage");
        myImage.setSrc("data:image/jpeg;base64," + imageData);

        mltext.getText(onSuccess, onFail, {
            imgType: 4,
            imgSrc: imageData
        });
        // for imgType Use 0,1,2,3 or 4
        function onSuccess(recognizedText) {
            //var element = document.getElementById('pp');
            //element.innerHTML=recognizedText.blocks.blocktext;
            //Use above two lines to show recognizedText in html
            console.log(recognizedText);
            alert(recognizedText.blocks.blocktext);
        }

        function onFail(message) {
            alert('Failed because: ' + message);
        }
    },

    onPhotoURISuccess: function(imageURI) {
        var myImage = oView.byId("myImage");
        myImage.setSrc(imageURI);

        mltext.getText(onSuccess, onFail, {
            imgType: 0,
            imgSrc: imageURI
        });
        // for imgType Use 0,1,2,3 or 4
        function onSuccess(recognizedText) {
            //var element = document.getElementById('pp');
            //element.innerHTML=recognizedText.blocks.blocktext;
            //Use above two lines to show recognizedText in html
            console.log(recognizedText);
            alert(recognizedText.blocks.blocktext);
        }

        function onFail(message) {
            alert('Failed because: ' + message);
        }
    },

    onFail: function(message) {
        console.log("Failed because: " + message);
    },

    getPhoto: function() {
        var oNav = navigator.camera;
        oNav.getPicture(this.onPhotoURISuccess, this.onFail, {
            quality: 100,
            destinationType: oNav.DestinationType.FILE_URI,
            sourceType: oNav.PictureSourceType.PHOTOLIBRARY
        });

    },

    capturePhoto: function() {
        var oNav = navigator.camera;
        oNav.getPicture(this.onPhotoDataSuccess, this.onFail, {
            quality: 100,
            destinationType: oNav.DestinationType.DATA_URL
        });
    }
});

Complete code is here.

And finally that’s all that I would like to cover.