Call Leonardo ML Foundation API from my SmartPhone via SAP Cloud Platform Functions.

The original post is here.(Japanese language)

I tried creating SNS chatbot using SAP Cloud Platform Functions and SAP Leonardo ML Foundation API, so I will show you how to make it.

You will know..

• How to use SAP Leonardo ML Foundation API.
• You can call SAP Leonardo ML Foundation API from Node.js.

Estimated time

• 90 minutes

---

Appendix: About LINE

LINE is a popular SNS tool in the Asian region, especially in Japan.
The number of users in Japan is about 76 million. The daily active user (DAU) boasts a very high figure of 85%, both number of users and frequency of use.

Appendix: What is the LINE Messaging API?

It is an API that realizes two-way communication with users through LINE’s account.

It is possible to develop interactive Bot application using LINE talk screen.
This API has already been used in many corporate LINE accounts, such as courier delivery redelivery acceptance and new year greeting creation.

Appendix: What is SAP Cloud Platform Functions?

It is a program execution environment of serverless architecture previously announced by SAP.
You can run the program created with Node.js.

---

Outline of the procedure

1. Try Leonardo ML Foundation API with SAP API Business Hub.
2. Create bot channel (bot account).
   Log in with a LINE account and create and set up a BOT account.
3. Activate service of Functions.
   Access the SAP Cloud Platform CF Trial Europe (Frankfurt) environment and make initial settings for using Functions.
4. Create a Node.js program.
   We will create a program to receive image messages from SNS and reply.
5. Test on actual smartphone.

 

1.Try Leonardo Foundation ML API with SAP API Business Hub.

With the SAP API Business Hub, it is possible to easily try the Leonardo Machine Learning API.

In Leonardo ML Foundation, you can try

1. Use pre-trained AI.
2. Use the AI that the user has re-trained.

In this time we will call up the pre-trained image classification API.

Open the API Business Hub Image Classification API page in your browser and click the Try out button of the API you want to try.

Select an image file and try it.

The result of AI judging the image will be returned.

By pressing the Code Snippet button on the API Business Hub, you can refer to the code snippet that calls the API from various programming languages.

In the following procedure, API Key is used when calling Leonardo ML API from Node.js. Look at the screen of the snippet and let’s keep an API Key.

 

2.Create bot channel (bot account).

We will create a channel as per the official page of LINE.
Please refer to the my previous article for setting method and setting place.
I created a channel named “画像認識くん (means Image recognition man)” this time.

Perform basic setting of the channel. We change the setting in red.

 

3.Activate service of Functions.

Functions are only available in the SAP CP CF Trial Europe (Frankfurt).
You also need to activate the beta service.

First, create a new subaccount that the Beta service can use.

Next, activate the Functions service of the trial2 subaccount.

Create an instance of Functions.

 

4.Create a Node.js program.

Access the Functions dashboard and create a program.

I made the program name “firstmlbot”.

Copy the following code and use it.
Please change **********  place to access token for the channel of LINE BOT.
Please change YOUR_API_KEY place to API Key of Leonardo ML API.

■index.js

var request = require("request");
var https = require("https");

module.exports = {
    handler: function(event, context) {

        if (event.data) {
            if (event.data.events[0].type == "message") {
                if (event.data.events[0].message.type == "image") {
                    var imageid = event.data.events[0].message.id;

                    // get image from below link
                    //https://api.line.me/v2/bot/message/{imageid}/content
                    var options = {
                        method: 'GET',
                        uri: 'https://api.line.me/v2/bot/message/' + imageid + '/content',
                        encoding: null,
                        auth: {
                            bearer: "*********************" // LINE BOT Access Token
                        }
                    };

                    request(options, function(error, response, body) {

                        // get binary image
                        var binaryimage = new Buffer(body);

                        // request to SAP ML Server
                        var boundary = createBoundary();
                        let optionstosap = {
                            host: "sandbox.api.sap.com",
                            port: 443,
                            path: "/ml/imageclassification/classification",
                            method: "POST",
                            headers: {
                                "APIKey": "YOUR_API_KEY",
                                "Content-Type": "multipart/form-data; boundary=" + boundary
                            }
                        };

                        var reqtosap = https.request(optionstosap, function(res) {
                            var data2 = '';
                            res.setEncoding("utf8");

                            res.on("data", (chunk) => {

                                data2 += chunk;

                                var resultarr = JSON.parse(data2).predictions[0].results;

                                var columns = [];

                                for (var k = 0; k < resultarr.length; k++) {

                                    var column = {};
                                    column.title = resultarr[k].label.substring(0, 35);
                                    column.text = Math.round(resultarr[k].score * 100 * 10) / 10 + "%";
                                    var actions = [];
                                    var action = {};
                                    action.type = "uri";
                                    action.label = "Google翻訳";
                                    action.uri = "https://translate.google.co.jp/?hl=ja#en/ja/" + encodeURIComponent(resultarr[k].label);
                                    actions.push(action);
                                    column.actions = actions;
                                    columns.push(column);
                                }

                                var options2 = {
                                    method: 'POST',
                                    uri: 'https://api.line.me/v2/bot/message/reply',
                                    body: {
                                        replyToken: event.data.events[0].replyToken,
                                        messages: [{
                                            type: "template",
                                            altText: "Image classification result.",
                                            template: {
                                                type: "carousel",
                                                columns: columns
                                            }
                                        }]
                                    },
                                    auth: {
                                        bearer: "*********************" // LINE BOT Access Token
                                    },
                                    json: true
                                };
                                request(options2, function(err, res, body) {
                                    console.log(JSON.stringify(res));
                                });


                            });

                            reqtosap.on('end', function() {
                                console.log('data;', data2);
                                res.end();
                            });

                        });

                        reqtosap.on("error", function(e) {
                            console.error(e.message);
                        });


                        var buffer = unicode2buffer(
                            '--' + boundary + '\r\n' + 'Content-Disposition: form-data; name="files"; filename="myimage.png"\r\n' +
                            'Content-Type: image/png\r\n\r\n'
                        );

                        var buffer = appendBuffer(buffer,
                            binaryimage
                        );

                        var buffer = appendBuffer(buffer,
                            unicode2buffer(
                                '\r\n' + '--' + boundary + '--'
                            )
                        );

                        reqtosap.write(Buffer.from(buffer));
                        reqtosap.end();

                    });
                }
            }
        }


        console.log('event data ' + JSON.stringify(event.data));
        return 'hello world from a function!';
    }
}


function createBoundary() {
    var multipartChars = "-_1234567890abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";
    var length = 30 + Math.floor(Math.random() * 10);
    var boundary = "---------------------------";
    for (var i = 0; i < length; i++) {
        boundary += multipartChars.charAt(Math.floor(Math.random() * multipartChars.length));
    }
    return boundary;
}

function unicode2buffer(str) {

    var n = str.length,
        idx = -1,
        byteLength = 512,
        bytes = new Uint8Array(byteLength),
        i, c, _bytes;

    for (i = 0; i < n; ++i) {
        c = str.charCodeAt(i);
        if (c <= 0x7F) {
            bytes[++idx] = c;
        } else if (c <= 0x7FF) {
            bytes[++idx] = 0xC0 | (c >>> 6);
            bytes[++idx] = 0x80 | (c & 0x3F);
        } else if (c <= 0xFFFF) {
            bytes[++idx] = 0xE0 | (c >>> 12);
            bytes[++idx] = 0x80 | ((c >>> 6) & 0x3F);
            bytes[++idx] = 0x80 | (c & 0x3F);
        } else {
            bytes[++idx] = 0xF0 | (c >>> 18);
            bytes[++idx] = 0x80 | ((c >>> 12) & 0x3F);
            bytes[++idx] = 0x80 | ((c >>> 6) & 0x3F);
            bytes[++idx] = 0x80 | (c & 0x3F);
        }
        if (byteLength - idx <= 4) {
            _bytes = bytes;
            byteLength *= 2;
            bytes = new Uint8Array(byteLength);
            bytes.set(_bytes);
        }
    }
    idx++;

    var result = new Uint8Array(idx);
    result.set(bytes.subarray(0, idx), 0);

    return result.buffer;
}

function appendBuffer(buf1, buf2) {
    var uint8array = new Uint8Array(buf1.byteLength + buf2.byteLength);
    uint8array.set(new Uint8Array(buf1), 0);
    uint8array.set(new Uint8Array(buf2), buf1.byteLength);
    return uint8array.buffer;
}

■Dependencies

{
  "dependencies": {
    "request": "*"
  }
} 

(Remind) The access token can be confirmed on the basic setting screen of the LINE BOT channel.

(Remind) API Key can be confirmed in Code Snippet of SAP API Business Hub.

After editing the source code, press the Save and Deploy button in the upper right corner of the dashboard to save.

Next, the trigger setting. This time, we use HTTP trigger.

---

Appendix: Triggers available in Functions

---

Copy the Trigger URL of Functions and paste it in the Webhook URL on the channel setting screen of LINE Developers.

This completes the development. We will do a test.

 

5.Test on actual smartphone

Read the QR code of the channel setting screen of LINE Developers on the smartphone and add account.

Sending an image to BOT will determine what image it is.

Almost OK.

OK!!

No.. it’s my room.

Nooooo, it’s my room!!

By loading the following QR code with your smartphone, you can easily experience the LINE BOT created this time.(If you have LINE account.)

You can also invite this LINE BOT to group chat for use.
Please invite them to LINE group chat and play together with people you know.

When using Leonardo ML image classification function in your business, you should use AI that has been re-training, not trained AI.

As for Leonardo ML’s re-training, I would like to write it as an article.