Skip to Content

The original post is here.(Japanese language)

I tried creating SNS chatbot using SAP Cloud Platform Functions and SAP Leonardo ML Foundation API, so I will show you how to make it.

You will know..

  • How to use SAP Leonardo ML Foundation API.
  • You can call SAP Leonardo ML Foundation API from Node.js.

Estimated time

  • 90 minutes

Appendix: About LINE

LINE is a popular SNS tool in the Asian region, especially in Japan.
The number of users in Japan is about 76 million. The daily active user (DAU) boasts a very high figure of 85%, both number of users and frequency of use.

Appendix: What is the LINE Messaging API?

It is an API that realizes two-way communication with users through LINE’s account.

It is possible to develop interactive Bot application using LINE talk screen.
This API has already been used in many corporate LINE accounts, such as courier delivery redelivery acceptance and new year greeting creation.

Appendix: What is SAP Cloud Platform Functions?

It is a program execution environment of serverless architecture previously announced by SAP.
You can run the program created with Node.js.


Outline of the procedure

  1. Try Leonardo ML Foundation API with SAP API Business Hub.
  2. Create bot channel (bot account).
    Log in with a LINE account and create and set up a BOT account.
  3. Activate service of Functions.
    Access the SAP Cloud Platform CF Trial Europe (Frankfurt) environment and make initial settings for using Functions.
  4. Create a Node.js program.
    We will create a program to receive image messages from SNS and reply.
  5. Test on actual smartphone.

 

1.Try Leonardo Foundation ML API with SAP API Business Hub.

With the SAP API Business Hub, it is possible to easily try the Leonardo Machine Learning API.

In Leonardo ML Foundation, you can try

  1. Use pre-trained AI.
  2. Use the AI that the user has re-trained.

In this time we will call up the pre-trained image classification API.

Open the API Business Hub Image Classification API page in your browser and click the Try out button of the API you want to try.

Select an image file and try it.

The result of AI judging the image will be returned.

By pressing the Code Snippet button on the API Business Hub, you can refer to the code snippet that calls the API from various programming languages.

In the following procedure, API Key is used when calling Leonardo ML API from Node.js. Look at the screen of the snippet and let’s keep an API Key.

 

2.Create bot channel (bot account).

We will create a channel as per the official page of LINE.
Please refer to the my previous article for setting method and setting place.
I created a channel named “画像認識くん (means Image recognition man)” this time.

Perform basic setting of the channel. We change the setting in red.

 

3.Activate service of Functions.

Functions are only available in the SAP CP CF Trial Europe (Frankfurt).
You also need to activate the beta service.

First, create a new subaccount that the Beta service can use.

Next, activate the Functions service of the trial2 subaccount.

Create an instance of Functions.

 

4.Create a Node.js program.

Access the Functions dashboard and create a program.

I made the program name “firstmlbot”.

Copy the following code and use it.
Please change **********  place to access token for the channel of LINE BOT.
Please change YOUR_API_KEY place to API Key of Leonardo ML API.

■index.js

var request = require("request");
var https = require("https");

module.exports = {
    handler: function(event, context) {

        if (event.data) {
            if (event.data.events[0].type == "message") {
                if (event.data.events[0].message.type == "image") {
                    var imageid = event.data.events[0].message.id;

                    // get image from below link
                    //https://api.line.me/v2/bot/message/{imageid}/content
                    var options = {
                        method: 'GET',
                        uri: 'https://api.line.me/v2/bot/message/' + imageid + '/content',
                        encoding: null,
                        auth: {
                            bearer: "*********************" // LINE BOT Access Token
                        }
                    };

                    request(options, function(error, response, body) {

                        // get binary image
                        var binaryimage = new Buffer(body);

                        // request to SAP ML Server
                        var boundary = createBoundary();
                        let optionstosap = {
                            host: "sandbox.api.sap.com",
                            port: 443,
                            path: "/ml/imageclassification/classification",
                            method: "POST",
                            headers: {
                                "APIKey": "YOUR_API_KEY",
                                "Content-Type": "multipart/form-data; boundary=" + boundary
                            }
                        };

                        var reqtosap = https.request(optionstosap, function(res) {
                            var data2 = '';
                            res.setEncoding("utf8");

                            res.on("data", (chunk) => {

                                data2 += chunk;

                                var resultarr = JSON.parse(data2).predictions[0].results;

                                var columns = [];

                                for (var k = 0; k < resultarr.length; k++) {

                                    var column = {};
                                    column.title = resultarr[k].label.substring(0, 35);
                                    column.text = Math.round(resultarr[k].score * 100 * 10) / 10 + "%";
                                    var actions = [];
                                    var action = {};
                                    action.type = "uri";
                                    action.label = "Google翻訳";
                                    action.uri = "https://translate.google.co.jp/?hl=ja#en/ja/" + encodeURIComponent(resultarr[k].label);
                                    actions.push(action);
                                    column.actions = actions;
                                    columns.push(column);
                                }

                                var options2 = {
                                    method: 'POST',
                                    uri: 'https://api.line.me/v2/bot/message/reply',
                                    body: {
                                        replyToken: event.data.events[0].replyToken,
                                        messages: [{
                                            type: "template",
                                            altText: "Image classification result.",
                                            template: {
                                                type: "carousel",
                                                columns: columns
                                            }
                                        }]
                                    },
                                    auth: {
                                        bearer: "*********************" // LINE BOT Access Token
                                    },
                                    json: true
                                };
                                request(options2, function(err, res, body) {
                                    console.log(JSON.stringify(res));
                                });


                            });

                            reqtosap.on('end', function() {
                                console.log('data;', data2);
                                res.end();
                            });

                        });

                        reqtosap.on("error", function(e) {
                            console.error(e.message);
                        });


                        var buffer = unicode2buffer(
                            '--' + boundary + '\r\n' + 'Content-Disposition: form-data; name="files"; filename="myimage.png"\r\n' +
                            'Content-Type: image/png\r\n\r\n'
                        );

                        var buffer = appendBuffer(buffer,
                            binaryimage
                        );

                        var buffer = appendBuffer(buffer,
                            unicode2buffer(
                                '\r\n' + '--' + boundary + '--'
                            )
                        );

                        reqtosap.write(Buffer.from(buffer));
                        reqtosap.end();

                    });
                }
            }
        }


        console.log('event data ' + JSON.stringify(event.data));
        return 'hello world from a function!';
    }
}


function createBoundary() {
    var multipartChars = "-_1234567890abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";
    var length = 30 + Math.floor(Math.random() * 10);
    var boundary = "---------------------------";
    for (var i = 0; i < length; i++) {
        boundary += multipartChars.charAt(Math.floor(Math.random() * multipartChars.length));
    }
    return boundary;
}

function unicode2buffer(str) {

    var n = str.length,
        idx = -1,
        byteLength = 512,
        bytes = new Uint8Array(byteLength),
        i, c, _bytes;

    for (i = 0; i < n; ++i) {
        c = str.charCodeAt(i);
        if (c <= 0x7F) {
            bytes[++idx] = c;
        } else if (c <= 0x7FF) {
            bytes[++idx] = 0xC0 | (c >>> 6);
            bytes[++idx] = 0x80 | (c & 0x3F);
        } else if (c <= 0xFFFF) {
            bytes[++idx] = 0xE0 | (c >>> 12);
            bytes[++idx] = 0x80 | ((c >>> 6) & 0x3F);
            bytes[++idx] = 0x80 | (c & 0x3F);
        } else {
            bytes[++idx] = 0xF0 | (c >>> 18);
            bytes[++idx] = 0x80 | ((c >>> 12) & 0x3F);
            bytes[++idx] = 0x80 | ((c >>> 6) & 0x3F);
            bytes[++idx] = 0x80 | (c & 0x3F);
        }
        if (byteLength - idx <= 4) {
            _bytes = bytes;
            byteLength *= 2;
            bytes = new Uint8Array(byteLength);
            bytes.set(_bytes);
        }
    }
    idx++;

    var result = new Uint8Array(idx);
    result.set(bytes.subarray(0, idx), 0);

    return result.buffer;
}

function appendBuffer(buf1, buf2) {
    var uint8array = new Uint8Array(buf1.byteLength + buf2.byteLength);
    uint8array.set(new Uint8Array(buf1), 0);
    uint8array.set(new Uint8Array(buf2), buf1.byteLength);
    return uint8array.buffer;
}

■Dependencies

{
  "dependencies": {
    "request": "*"
  }
} 

(Remind) The access token can be confirmed on the basic setting screen of the LINE BOT channel.

(Remind) API Key can be confirmed in Code Snippet of SAP API Business Hub.

After editing the source code, press the Save and Deploy button in the upper right corner of the dashboard to save.

Next, the trigger setting. This time, we use HTTP trigger.


Appendix: Triggers available in Functions

Trigger Description
HTTP It issues a URL and fires when an HTTP request is made for that URL.
Timer Processing can be executed at regular intervals.
Event Combined with SAP Enterprise Messaging, you can operate the application with the event sent from outside as a trigger.

Copy the Trigger URL of Functions and paste it in the Webhook URL on the channel setting screen of LINE Developers.

This completes the development. We will do a test.

 

5.Test on actual smartphone

Read the QR code of the channel setting screen of LINE Developers on the smartphone and add account.

Sending an image to BOT will determine what image it is.

Almost OK.

OK!!

No.. it’s my room.

Nooooo, it’s my room!!

By loading the following QR code with your smartphone, you can easily experience the LINE BOT created this time.(If you have LINE account.)

You can also invite this LINE BOT to group chat for use.
Please invite them to LINE group chat and play together with people you know.

When using Leonardo ML image classification function in your business, you should use AI that has been re-training, not trained AI.

As for Leonardo ML’s re-training, I would like to write it as an article.

To report this post you need to login first.

8 Comments

You must be Logged on to comment or reply to a post.

  1. Moya Watson

    Cool share Yasayuki! Thanks for taking the time to write it both in Japanese and English!

    We’re tweeting it out via @sapcp – feel free to tag us with future posts, and thanks for contributing!

    (0) 
  2. Yatsea Li

    Hi yasuyuki-san,

    Thanks for an informative blog about integrating Line and Leonardo ML with SCP Function.

    Just wonder if you have tried SCP function for Facebook Messenger webhook other than Line.

    I have tried it with no luck.

    No query parameters(token etc) for validation from facebook are received in the event(event.data) object of the function, therefore the webhook validation doesn’t go through.

    Btw, FB messenger webhook works fine with AWS Lambda, the validation token available as event.params.querystring

    Kind Regards, Yatsea

    (0) 
    1. yasuyuki uno Post author

      Hello Yatsea,

      Since I saw your comment, I just created Facebook Bot for the first time.

      Although I was having difficulty, I was able to develop it in the end.

       

      Here is the source code. ( Please change according to your setting at “// PLEASE_CHANGE_HERE” commented line.)
      Sorry for un-beautiful source code.

      var request = require("request");
      module.exports = { 
       handler: function (event, context) { 
           
           var req = event.extensions.request;
           var res = event.extensions.response;
           var webhook_callback_token = "<YOUR_WEBHOOK_CALLBACK_TOKEN>";  // PLEASE_CHANGE_HERE
           console.log(event.extensions.request.method);
           
           if (req.method == "GET"){
              if (req.query['hub.verify_token'] === webhook_callback_token ) { 
                  res.send(req.query['hub.challenge']);
              }
              res.send('Error, wrong token');
           }
           else if (req.method == "POST"){
               // POST
               console.log(event.data);
               console.log("-----");
               
                let messaging_events = event.data.entry[0].messaging;
                for (let i = 0; i < messaging_events.length; i++) {
                  let ev = event.data.entry[0].messaging[i];
                  let sender = ev.sender.id;
                  if (ev.postback) {
                    let text = JSON.stringify(ev.postback);
                    console.log(ev);
                    if(ev.postback.payload === "Greeting") {
                      welcomeTextMessage(sender);
                    } else {
                      console.log("error");
                    }
                    continue;
                  }
                  if (ev.message && ev.message.text) {
                    let text = ev.message.text;
                    
                    sendTextMessage(sender, "Text received!! echo: " + text.substring(0, 200));
                  }
                }
                res.sendStatus(200);
           }
        } 
       }
       
      function sendTextMessage(sender, text) {
        let messageData = { text:text };
        request({
          url: 'https://graph.facebook.com/v2.6/me/messages',
          qs: {access_token:"<YOUR_PAGE_ACCESS_TOKEN>"},   // PLEASE_CHANGE_HERE
          method: 'POST',
          json: {
              recipient: {id:sender},
            message: messageData,
          }
        }, 
        function(error, response, body) {
          if (error) {
            console.log('Error sending messages: ', error);
          } else if (response.body.error) {
            console.log('Error: ', response.body.error);
          }
        });
      }
      
      function welcomeTextMessage(sender) { // Welcome message on push start button.
        let messageData = { text:"Welcome to test chatbot." };
        request({
          url: 'https://graph.facebook.com/v2.6/me/messages',
          qs: {access_token:"<YOUR_PAGE_ACCESS_TOKEN>"},   // PLEASE_CHANGE_HERE
          method: 'POST',
          json: {
            recipient: {id:sender},
            message: messageData,
          }
        }, 
        function(error, response, body) {
          if (error) {
              console.log('Error sending messages: ', error);
          } else if (response.body.error) {
              console.log('Error: ', response.body.error);
            }
        });
      }

       

      Hope your help,

      Yasuyuki

       

      (1) 
      1. Yatsea Li

        Hi Yasuyuki,

        Excellent. So we can have access to the request and response object through event.extensions.
        There is little document about this on the official online help document.

        I tried to look into event object with console.log on execution log, but having some problem to retrieve the execution logs of the functions on the tool.

        Thank you very much for sharing.

        Kind Regards, Yatsea

        (0) 
        1. yasuyuki uno Post author

          Yes, we can full access to the request and response objects through event.extensions.

           

          Attention: If you want to get the request.body informations….

          module.exports = { 
           handler: function (event, context) { 
               var req = event.extensions.request;
               var res = event.extensions.response;
               
               if (req.method == "GET"){
          
                   res.send("This is get request.");  
               }
               else{
                   // req.body is buffer object
                   console.log(req.body);
                   // Output: <Buffer 54 68 69 73 20 69 .....
                   console.log(req.body.foo);
                   // Output: undefined
          
          
                   // If you want to access body information, use this way.
                   var parsedbody = JSON.parse(req.body);
                   console.log(parsedbody);
                   // Output: { foo: 'bar', somecolumn: 'baz'}
                   console.log(parsedbody.foo);
                   // Output: bar
          
                   res.send("This is post request.");  
               }
              
            } 
           }

          Hope your help,

          Yasuyuki

          (0) 
  3. Yatsea Li

    Hi Yasuyuki,

    I have your code ported  for Facebook Messenger as below. Thanks for sharing again.

    let request = require("request");
    let https = require("https");
    
    //configuration
    let PAGE_TOKEN = "Place the page token of your Facebook App";
    let VERIFY_TOKEN = "Place your token here";
    let API_KEY = "Place your SAP Leonard ML key here";
    
    module.exports = {
        handler: function (event, context) {
    
            var req = event.extensions.request;
            var res = event.extensions.response;
            console.log(JSON.stringify(event.extensions.request.method));
    
            //GET: Token verification for FB webhook
            if (req.method == "GET") {
                if (req.query['hub.verify_token'] === VERIFY_TOKEN) {
                    res.send(req.query['hub.challenge']);
                }
                res.send('Error, wrong verify token');
            } else if (req.method == "POST") {
                // POST
                console.log(event.data);
                console.log("-----");
                handleMessage(event.data, res);
            }
        }
    }
    
    //General wrapper of calling facebook api to send the message.
    function callSendAPI(sender_psid, response) {
        // Construct the message body
        let request_body = {
            "recipient": {
                "id": sender_psid
            },
            "message": response
        }
    
        request({
            "uri": "https://graph.facebook.com/me/messages",
            "qs": {
                "access_token": PAGE_TOKEN
            },
            "method": "POST",
            "json": request_body
        }, (err, res, body) => {
            if (!err && res.statusCode === 200) {
                console.log('message sent!');
            } else {
                console.log('HTTP Response Status Code:', res && res.statusCode);
                console.log('Body:', JSON.stringify(body));
                console.error("Unable to send message:" + err);
                if (body && body.error && body.error.message) {
                    let msg = `${body.error.message}`;
                    console.error("Unable to send message:" + msg);
                    sendTextMessage(sender_psid, 'An error has occured.');
                }
            }
        });
    }
    
    //send a text message reply to the sender with the given text
    function sendTextMessage(sender_psid, text) {
        let response = {
            text: text
        };
        callSendAPI(sender_psid, response);
    }
    
    //The entry point of facebook message handling
    function handleMessage(body, res) {
        // Check the webhook event is from a Page subscription
        if (body.object === 'page') {
            body.entry.forEach(function (entry) {
                // Gets the body of the webhook event
                let webhook_event = entry.messaging[0];
                console.log(webhook_event);
    
                // Get the sender PSID
                let sender_psid = webhook_event.sender.id;
                console.log('Sender ID: ' + sender_psid);
    
                // Check if the event is a message or postback and
                // pass the event to the appropriate handler function
                if (webhook_event.message) {
                    handleTextAndAttachmentMessage(sender_psid, webhook_event.message);
                } else if (webhook_event.postback) {
                    handlePostback(sender_psid, webhook_event.postback);
                }
            });
            // Return a '200 OK' response to all events
            res.status(200).send('EVENT_RECEIVED');
    
        } else {
            // Return a '404 Not Found' if event is not from a page subscription
            res.sendStatus(404);
        }
    }
    
    function handlePostback(sender_psid, received_postback) {
        let response;
        // Get the payload for the postback
        let payload = received_postback.payload;
    
        //right now the only postback is Get Started
        //send a welcoem message.
        sendTextMessage(sender_psid, "Hello, welcome to the Image Classifer Bot powered by SAP Leonardo Machine Learning Foundation and SCP Function as a Service. Now you can send me a image.");
    }
    
    function handleTextAndAttachmentMessage(sender_psid, received_message) {
        if (received_message.attachments) {
            console.log('attachment received');
            console.log(received_message.attachments[0].payload);
            received_message.attachments.forEach(element => {
                if (element.type === 'image') {
                    handleImageMessage(sender_psid, element.payload.url);
                }
            });
        } else if (received_message.text) {
            sendTextMessage(sender_psid, `Please send me a photo. I will help you to classify it. Thanks.`);
        }
    }
    
    let ListTemplate = {
        "attachment": {
            "type": "template",
            "payload": {
                "template_type": "list",
                "top_element_style": "large",
                "elements": []
            }
        }
    };
    
    let ElementTemplate = `{
      "title": "",
      "image_url": "",
      "subtitle": "",
      "buttons": [
          {
              "title": "Google Translate",
              "type": "web_url",
              "url": "",
              "messenger_extensions": true,
              "webview_height_ratio": "tall",
              "fallback_url": ""
          }
      ]
    }`;
    
    function handleImageMessage(sender_psid, image_url) {
        //sendTextMessage(sender_psid, 'Image received.' + image_url);
        console.log('Image received: ' + image_url);
        let options = {
            method: 'GET',
            encoding: null,
            uri: image_url,
        };
    
        request(options, function (error, response, body) {
            // get binary image
            if (error) {
                console.log('error occured:' + error);
                Return;
            }
    
            console.log('Image downloaded as binary data.');
            var binaryimage = new Buffer(body);
            // request to SAP ML Server
            var boundary = createBoundary();
            console.log('Boundary created: ' + boundary);
            let optionstosap = {
                host: "sandbox.api.sap.com",
                port: 443,
                path: "/ml/imageclassification/classification",
                method: "POST",
                headers: {
                    "APIKey": API_KEY,
                    "Content-Type": "multipart/form-data; boundary=" + boundary
                }
            };
    
            /****/
            var reqtosap = https.request(optionstosap, function (res) {
                var data2 = '';
                res.setEncoding("utf8");
                res.on("data", (chunk) => {
                    data2 += chunk;
                    console.log('classification result: ' + data2);
                    let result_obj = JSON.parse(data2);
                    if(result_obj && result_obj.error)
                    {
                        console.error(`An error has occurred. 
    err code: ${result_obj.error.code}. 
    err message: ${result_obj.error.message}.`);
                        sendTextMessage(sender_psid, 'An error has occurred');
                        return;
                    }
    
                    var resultarr = result_obj.predictions[0].results;
                    let result = ListTemplate;
                    result.attachment.payload.elements = [];
                    let count = resultarr.length <= 4? resultarr.length:4;
                    for (var k = 0; k < count; k++) {
                        let element = JSON.parse(ElementTemplate);
                        element.title = resultarr[k].label.substring(0, 35);
                        element.subtitle = Math.round(resultarr[k].score * 100 * 10) / 10 + "%";
                        element.image_url = image_url;
                        element.buttons[0].url = "https://translate.google.co.jp/?hl=ja#en/ja/" + encodeURIComponent(resultarr[k].label);
                        result.attachment.payload.elements.push(element);
                    }
                    console.log(JSON.stringify(result));
                    callSendAPI(sender_psid, result);
                });
    
                reqtosap.on('end', function () {
                    console.log('data: ', data2);
                    res.end();
                });
    
            });
    
            reqtosap.on("error", function (e) {
                console.error(e.message);
            });
    
            let buffer = unicode2buffer(
                '--' + boundary + '\r\n' + 'Content-Disposition: form-data; name="files"; filename="myimage.jpg"\r\n' +
                'Content-Type: image/jpg\r\n\r\n'
            );
    
            let buffer2 = appendBuffer(buffer,
                binaryimage
            );
    
            let buffer3 = appendBuffer(buffer2,
                unicode2buffer(
                    '\r\n' + '--' + boundary + '--'
                )
            );
    
            reqtosap.write(Buffer.from(buffer3));
            reqtosap.end();
            /***/
        });
    
        function createBoundary() {
            var multipartChars = "-_1234567890abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";
            var length = 30 + Math.floor(Math.random() * 10);
            var boundary = "---------------------------";
            for (var i = 0; i < length; i++) {
                boundary += multipartChars.charAt(Math.floor(Math.random() * multipartChars.length));
            }
            return boundary;
        }
    
        function unicode2buffer(str) {
    
            var n = str.length,
                idx = -1,
                byteLength = 512,
                bytes = new Uint8Array(byteLength),
                i, c, _bytes;
    
            for (i = 0; i < n; ++i) {
                c = str.charCodeAt(i);
                if (c <= 0x7F) {
                    bytes[++idx] = c;
                } else if (c <= 0x7FF) {
                    bytes[++idx] = 0xC0 | (c >>> 6);
                    bytes[++idx] = 0x80 | (c & 0x3F);
                } else if (c <= 0xFFFF) {
                    bytes[++idx] = 0xE0 | (c >>> 12);
                    bytes[++idx] = 0x80 | ((c >>> 6) & 0x3F);
                    bytes[++idx] = 0x80 | (c & 0x3F);
                } else {
                    bytes[++idx] = 0xF0 | (c >>> 18);
                    bytes[++idx] = 0x80 | ((c >>> 12) & 0x3F);
                    bytes[++idx] = 0x80 | ((c >>> 6) & 0x3F);
                    bytes[++idx] = 0x80 | (c & 0x3F);
                }
                if (byteLength - idx <= 4) {
                    _bytes = bytes;
                    byteLength *= 2;
                    bytes = new Uint8Array(byteLength);
                    bytes.set(_bytes);
                }
            }
            idx++;
    
            var result = new Uint8Array(idx);
            result.set(bytes.subarray(0, idx), 0);
    
            return result.buffer;
        }
    
        function appendBuffer(buf1, buf2) {
            var uint8array = new Uint8Array(buf1.byteLength + buf2.byteLength);
            uint8array.set(new Uint8Array(buf1), 0);
            uint8array.set(new Uint8Array(buf2), buf1.byteLength);
            return uint8array.buffer;
        }
    }
    

     

    Kind Regards, Yatsea

    (1) 

Leave a Reply