Skip to Content

In this blog post I will give a short overview about my business card crm. It is a proof of concept for using almost serverless technology to solve an old problem.

What is the application good for?

This application can be used to make a photo of a business card. The image will be analyzed by the Google Vision API and afterwards the text will be feed into the Google Named Entity Recognition Engine. Based on some look up tables and regular expressions typically attributes like first name, last name, phone number and email address are extracted. The data will be saved in the local storage of the device. If multiple clients like a smartphone and a desktop PC want to share business card data they can join together through a WebRTC session and data can be send through this connection.

Video tour

Architecture

Source: Manuel Blechschmidt, created with Signavio Process Editor

Example image

The image above shows a typical business card like you can collect on a lot of different occasions. Typically these are entered manually into a CRM system. More advanced systems do employ ocr recognition technology to automate parts of this error prone process.

Google offers an API for analyzing such images and get the text. This API can directly be consumed with JavaScript after creating an access key.

The code looks in OpenUI5 like this:

 

var _gApiLoaded = new Promise(function(resolve) {
 jQuery.sap.includeScript("https://apis.google.com/js/api.js", "gapi", function() {
     gapi.load("client", function() {
         gapi.client.setApiKey("<Insert your key here>");
         resolve();
     });
 });
}); // https://www.googleapis.com/auth/cloud-platform

_gApiLoaded.then(function() {
    // https://developers.google.com/discovery/v1/reference/apis/list Just hit execute to get a list
    gapi.client.load("https://vision.googleapis.com/$discovery/rest", "v1", function() {
        var reader = new FileReader();
        reader.readAsDataURL(oBlob);
        reader.onloadend = function() {
            var base64data = reader.result;
            gapi.client.vision.images.annotate({
                "requests": [{
                    "features": [{
                        "type": "TEXT_DETECTION"
                    }],
                    "image": {
                        "content": base64data.replace(/^data:image\/jpeg;base64,/, "")
                    }
                }]
            }).then(function(oResponse) {
                console.log({
                    "ImageBase64": base64data,
                    "OCRResult": oResponse.result
                });
            });
        };
    });

});

For the image above the result looks like this:

{
  "responses": [
    {
      "textAnnotations": [
        {
          "locale": "en",
          "description": "Incentergy\nGmbH\nManuel Blechschmidt\nCEO\nwww.incentergy.de\nPhone: +49 173 632 26 21\nMail: manuel.blechschmidt@incentergy.de\n",
          "boundingPoly": {
            "vertices": [
              {
                "x": 42,
                "y": 151
              },
              {
                "x": 901,
                "y": 151
              },
              {
                "x": 901,
                "y": 661
              },
              {
                "x": 42,
                "y": 661
              }
            ]
          }
        },
        {
          "description": "Incentergy", ...

Afterwards the whole text is send to google NER engine.

{
  "entities": [
    {
      "name": "CEO",
      "type": "PERSON",
      "metadata": {},
      "salience": 0.3584892,
      "mentions": [
        {
          "text": {
            "content": "CEO",
            "beginOffset": 36
          },
          "type": "COMMON"
        }
      ]
    },
    {
      "name": "Phone",
      "type": "CONSUMER_GOOD",
      "metadata": {},
      "salience": 0.2520285,
      "mentions": [
        {
          "text": {
            "content": "Phone",
            "beginOffset": 58
          },
          "type": "COMMON"
        }
      ]
    },
    {
      "name": "Incentergy GmbH Manuel Blechschmidt",
      "type": "PERSON",
      "metadata": {},
      "salience": 0.22339708,
      "mentions": [
        {
          "text": {
            "content": "Incentergy\nGmbH\nManuel Blechschmidt",
            "beginOffset": 0
          },
          "type": "PROPER"
        }
      ]
    }, ...

 

After letting Google do the hard lifting of OCR and NER the application find with some basic rules the correct strings.

Screenshots

SAP Web IDE

The application was completely develop in SAP Web IDE and published to github.

WebRTC

For synchronizing data between different devices the App can create a RTC session. For the details please consult the source code and the video shown above.

Summary

This applications shows what is possible with the newest HTML5 technology and machine learning including OpenUI5. It uses WebRTC, OpenUI5, LocalStorage, Google Vision API.

To report this post you need to login first.

1 Comment

You must be Logged on to comment or reply to a post.

Leave a Reply