Anki Cozmo meets SAP Cloud Platform – Part 3: Remote inspection

[nadine_gaertner]

This blog extends the first part where we created a basic robot controller, collected sensor data from the robot and set up a tiny app for remote controlling the robot.

Now, we add another feature to this IoT application to make it a little more interesting: We turn the Cozmo into a remote inspection device which takes pictures of its surroundings and displays them in the UI5 app. The general building blocks required to implement this feature, namely the picture taking via the Anki Cozmo API, the IoT service with its OData services, the and OData binding in UI5 are all out there and documented in various places. Here, we show how to bring it all together.

For this blog, we assume you have worked through the first part and have the initial version of the robot controller and the UI5 app set up and running.

Scenario

Again, we think of the Cozmo robot as an autonomous vehicle. As a remote inspection device, the robot can be sent to locations which might be dangerous, difficult or cumbersome to access for a human worker. The robot can then transmit pictures from the remote location via the SAP Cloud Platform. Once uploaded, the images can be assessed by a human expert -- or even by an algorithm -- to decide upon follow-up actions.

Outline

To realize the remote inspection we need to adjust three areas:

• Define a new message type for transmitting the pictures via IoT service


• Create new elements in the UI5 app for submitting the picture request to the robot and displaying the picture from the robot


• Enhance the robot controller so it takes & transmits a picture

Like in the previous parts of the blog, I explain step by step what needs to be adjusted in the code and show you the updated code snippet. Hopefully the explanations enable you to implement the changes yourself. If you are in a rush you can of course just copy the code - though possibly compromising the learning effect.

IoT service configuration

First we need to define a new IoT message type for the picture. The message type has one field of type "binary" since we are passing the picture as a base64-encoded string.

For creating the actual message type go to the Internet of Things cockpit. If you need more detailed instructions for navigating to and in the cockpit find them in the first part.



 

Next, you need to add the image message type to your cozmo device type. It appears it is not possible to edit an existing device type so you might need to create a new device type and a new device in the IoT service configuration.

Note down all the new IDs and update your config.py file in the Python part of the project accordingly.

 

UI5 enhancements

We extend the UI5 application with a button for triggering the robot to send a picture. Also, we add an image control for displaying the latest image from the robot.

After adding these two UI elements to the Cozmo.view.xml this file looks as follows. Note the event handler declaration of onTakePicture which we need to implement next.

<mvc:View controllerName="blogtest.controller.Cozmo" xmlns:html="http://www.w3.org/1999/xhtml" xmlns:mvc="sap.ui.core.mvc"

	displayBlock="true" xmlns="sap.m">

	<App>

		<Page title="Cozmo in the Internet of Things">

			<Panel class="sapUiResponsiveMargin" width="auto">

				<Button text="Move fork" press="onMoveFork" class="sapUiSmallMarginEnd"/>

				<Button text="Take picture" press="onTakePicture" class="sapUiSmallMarginEnd"/>

			</Panel>

			<Panel>

				<FlexBox id="imageBox" height="auto" alignItems="Start" justifyContent="Start">

					<Image id="cozmoView" width="70%"></Image>

				</FlexBox>

			</Panel>

		</Page>

	</App>

</mvc:View>

 

Previously, we triggered the Cozmo to move its fork by sending a POST to the IoT push service. The approach for triggering the picture taking is analogous. We just send a different action code. Action code 1 was for "moving the fork", so pick your number for "taking a picture". I pick 4 for the action code and refactor the previous code for reusability. Here is the excerpt of the Cozmo.controller.js with the event handlers for the two push buttons.

Cozmo.controller.js

triggerCozmoAction: function(actionCode) {

	var pushServiceUrl = "/iotmms/v1/api/http/push/<technical_ID_of_your_COZMO_device>";

	var messageObject = {

		"sender": "IoT application",

		"method": "ws",

		"messageType": "<technical_ID_of_your_TO_COZMO_message>",

		"messages": [{

			"ACTION": actionCode

		}]

	};



	var successHandler = function(oData, textStatus, jqXHR) {

		sap.m.MessageToast.show("Success");

	};



	this.doHttp("POST", pushServiceUrl, messageObject, successHandler);

},



onMoveFork: function() {

	this.triggerCozmoAction("1");

},



onTakePicture: function() {

	this.triggerCozmoAction("4");

	this.updateImage();

},

Note that besides sending the action code to the robot I already added a call to the yet to be defined method updateImage. Purpose of this new method is to retrieve the latest image from the message store of the IoT service and to then make this image the content of the "Image" UI5 element.

Sneaking in updateImage here is anticipatory in two ways: We have not implemented this method yet (but we will in a second) plus we are presuming that the robot has actually sent the image via IoT service after it was asked to do so (which we will take care of in the last section of this blog).

Admittedly, simply calling updateImage after triggering the picture taking action is not even correct -- and you will see this when you test the app later. What happens with this implementation is that the picture is only updated on the UI after triggering another picture snap. Reason is the time delay between triggering the robot to provide a picture and the actual availability of the picture in the IoT service. In a proper implementation you would need to move the call of updateImage and attach it for example to the onMessage event handler of the websocket. We skip this extra bit of asynchronism management for the time being to avoid additional code complexity.

Anyway, here are some hints for implementing updateImage:

• The IoT service provides an OData API that can be used to access tables and consume messages. Similar to the access of the IoT service push service we can access the OData API via the iotmms destination. For accessing the messages of the COZMO_IMAGE message type we concatenate the corresponding table name into the service URL. To understand the detailed syntax it might help to check out the IoT service documentation for an explanation of assembling the URL and the UI5 documentation for details on how to handle the OData model.


• Access to the OData model is asynchronous. After a read access to the data model the result can be retrieved by attaching to the request completed event.


• The UI5 image control accepts base64-encoded strings as content as long as they are prefixed with data:image/jpeg;base64,

Find my implementation below. Note that I am storing the retrieved image in a global variable imageBlob - which I declare in the index.html file as indicated in the snippet below. I am making use of this/that in order to access the Cozmo.controller out of the requestCompleted callback function. Feel free to implement a more elegant solution.

Cozmo.controller.js

updateImage: function(image) {

	var dataModel = new sap.ui.model.odata.v2.ODataModel("/iotmms/v1/api/http/app.svc/");

	dataModel.read("/T_IOT_<your_image_message_type>?$top=1&$select=C_IMAGE");

	var that = this;

	dataModel.attachRequestCompleted(function(oEvent) {

		var oResponse = JSON.parse(oEvent.getParameter("response").responseText);

		var blob = oResponse.d.results[oResponse.d.results.length - 1].C_IMAGE;

		imageBlob = "data:image/jpeg;base64," + blob;

		that.setImageContent(imageBlob);

	});

},



setImageContent: function(blob) {

	var oImg = this.byId("cozmoView");

	oImg.setSrc(blob);

},

index.html

<script>

	sap.ui.getCore().attachInit(function() {

		new sap.m.Shell({

			app: new sap.ui.core.ComponentContainer({

				height : "100%",

				name : "<your_project_name>"

			})

		}).placeAt("content");

	});

	var imageBlob;

</script>

 

Python enhancements

Finally, we need to take care of the robot controller and make sure it will react to the action trigger with action code 4 by taking and sending a picture via the new IoT service message type.

Recall that in the cozmo_websocket we defined a helper method send_position to send the robot's position data to the SAP Cloud Platform. Now we create a similar helper for sending the image. To send the image we need to specify the ID of the corresponding message type. For convenience I recommend adding this information to the config file.

cozmo_websocket.py

def send_image(self, messages):

    global ws, url, auth

    ws = create_connection(url, header = {"Authorization":auth})

    message = {"mode":"async","messageType":my_img_msg_type,"messages":messages}

    ws.send(json.dumps(message))

config.py

my_img_msg_type = "<your_image_message_id>"

Now, the websocket is prepared for sending images. For handling the "action code 4 request" we already have the on_message handler in place which we just enhance for the new action code.

For taking the actual picture with the Cozmo we first need to enable its camera image stream. Then the Cozmo robot is constantly taking pictures and we can access the latest_image from the Cozmo API.

In the snippets below find my updates to the cozmo_controller. Notes on my implementation:

• I add the general image stream enabling in the cozmo_program method


• I save the latest_image in a local file to then convert it into base64. The subsequent utf-8 encoding is necessary for the json format as used by the IoT service.

cozmo_controller.py

import base64

def cozmo_program(robot: cozmo.robot.Robot):

    global cozmo_instance, ws_communicator, keep_running

    cozmo_instance = robot

    cozmo_instance.camera.image_stream_enabled = True

    ws_communicator = websocket_scp()

    ws_communicator.start_listener(on_message)

    keep_running = True

    while True:

        run_cozmo()

def on_message(ws, message):

    data = json.loads(message)

    action = data["messages"][0]["ACTION"]

    if action == "1":

        cozmo_instance.set_lift_height(1.0, in_parallel=True)

        cozmo_instance.set_lift_height(0.0, in_parallel=True)

    if action == "4":

        take_picture()



def take_picture():

    print("Smile! taking a picture!")

    latest_image = cozmo_instance.world.latest_image

    if (latest_image is not None):

        latest_image.raw_image.save("latest_image.jpeg", "jpeg")

        with open("latest_image.jpeg", "rb") as imageFile:

            data = base64.b64encode(imageFile.read())

        data = str(data,'utf-8')

        messages = [{'IMAGE': data}]

        ws_communicator.send_image(messages)

 

Expected outcome

When you now run your UI5 app there is the new "take picture" button. When you push the button you will see impressions from Cozmo's perspective.

Congratulations, you are now ready for remote inspection!