Technology Blogs by SAP
Learn how to extend and personalize SAP applications. Follow the SAP technology blog for insights into SAP BTP, ABAP, SAP Analytics Cloud, SAP HANA, and more.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

From Arduino to SCP IoT Service

former_member250012
Explorer
‎01-05-2018 3:42 PM

From Arduino to SAP Cloud Platform Internet of Things Service


 

 

 



Introduction


After having heard a lot around Internet of Things (IoT), I wanted to have a better understanding on the end to end process behind such a project and I love the idea of building a small device from which we can get data or even interact with.

Since IoT is not really new anymore, you might have already found a lot of information on the net already.  So why yet another blog post?  Because I couldn't find a blog yet that was covering everything from beginning to end...

What I suggest is thus to achieve a very simple project covering the assembling of a devices using an Arduino board, capturing temperature data using a sensor, sending the data to the SAP Cloud Platform IoT Service and publish them through a mean that can be consumed from a device such as a mobile device.

The last bit that we will still miss is the development of an app that will consume the data from the SAP Cloud Platform which could be considered as a follow up to this tutorial.

 

 



Architecture description




The illustration above shows the complete flow of information from the Arduino board to the data consumer which could be a mobile device.  The data collected on the platform are also consumed by an HTML5 Application.  There is many possibilities to consumed the data once on the SAP Cloud Platform this diagram represent a solution among many others.

The scope of this tutorial goes beyond the board explaining how to connect a sensor to it and to develop the "Thing".  This tutorial doesn't go all the way down to the implementation of an application to consume the collected data, we will stop once the data get into the SAP Cloud Platform.

 



SAP Cloud Platform IoT Service Configuration



Enabling and accessing the SCP IoT Service


In your SAP Cloud Platform (SCP) go to the Services section and enter "iot" in the filter to display directly the "Internet of Things" service.



If the service has not been enable yet go into the service and press the Enable button.







Once the service enabled you can click on the link "Go to Service".



Deploying the Message and Management Service (MMS)


In the IoT service, go to the bottom of the page and click on the tile "Deploy the Message Management Service".



Provide your Account ID and your credentials for the Platform.









Granting IoT-MMS-User role


As mentioned in the interface: “Only users with the "IoT-MMS-User" role can access the Message Management Service dashboard. You can configure this authorization in the "Roles" section of the Java Application Dashboard.”  The next step is therefore to add that role to your user.

In SCP, go to the Security ► Authorizations.



In the "User" tab, enter your user id and press "Show Assignments".



Click on the "Assign" link above the table. Select the "iotmms" Application and the "IoT-MMS-User" Role.  Press the "Save" button.



We have now successfully configured the Platform to benefit from the IoT service and our user is capable to access it and work with it.


Defining a message type








NOTE: the timestamp that is sent should be an epoch date/time.  SCP will apply the time zone adjustment itself.  You send 23:30 epoch and 00:30 will be shown as data in IoT service…





Creating a device type










Press the "Create" button.







Adding a device


Some "sensible" data such as the security token and devices IDs are handled in this section.  Most people would scramble everything, I don't.  I prefer to have something pretty transparent and clear.  The project described in this tutorial as already moved to another level and almost everything of what have been described in this blog doesn't even exist anymore.









New OAuth access token for device "i063866_01" created. Token: 2b8358a867bdb7678b63c1973891b95



Device ID: 63c3ab7c-5cae-4611-b6ca-d9783db25cee

OAuth Token: 2b8358a867bdb7678b63c1973891b95

 

 



Testing/Sending information from and to the MMS








Scroll down and click on the tile "Messaging through HTTP".



 



Data Endpoint: https://iotmmsi063866trial.hanatrial.ondemand.com/com.sap.iotservices.mms/v1/api/http/data/d000-e00...

Message: {"mode":"sync","messageType":"m0t0y0p0e1","messages":[{"sensor":"sensor1","value":"20","timestamp":1413191650}]}

Modify the following info from the template (Data Endpoint and Message):

  • d000-e000-v000-i000-c000-e001 ► Device ID [63c3ab7c-5cae-4611-b6ca-d9783db25cee]

  • m0t0y0p0e1 ► Message Type ID [d6cdc3d436b9859af722]

  • {"sensor":"sensor1","value":"20","timestamp":1413191650} ► {"timestamp":1413191650,"celsiusTemp":"25.8"}




Ack Endpoint: https://iotmmsi063866trial.hanatrial.ondemand.com/com.sap.iotservices.mms/v1/api/http/ack/d000-e000-...

Modify the following info from the Ack Endpoint template:

  • d000-e000-v000-i000-c000-e001 ► Device ID [63c3ab7c-5cae-4611-b6ca-d9783db25cee]



Summary of the key information




Running the test






The very first time this request succeed you might get a 202 (The request has been accepted for processing, but the processing has not been completed).  Then the following ones will be a 200 (The request has succeeded).







The table “T_IOT_D6CDC3D436B9859AF722” got created automatically as soon as a first message was received on the Data Endpoint.  As you can see the last part of the table’s name corresponds to the Message Type ID.

The table “T_IOT_ACKSTORE” also gets created as soon as the Ack Endpoint receives its first message.



NOTE: We can see here that a time zone +2 is consider while on the chip not.   There,  the UTC/epoch time is used.



We now know that sending the message content as we shaped it, to the data endpoint reach the server properly.  We thus need to see how we can do that from the Arduino board…

 

 



Sending data from the Arduino Board


Installing the Arduino IDE


Download an install the IDE from the Arduino website.  There is also a web version available that will keep all your projects in a cloud and thus accessible from anywhere.  At the time of my experiments I used the desktop version but it shouldn't make any differences.



This screenshot shows on the left the edition area and on the right the serial monitor with can be used to troubleshoot your development.  In most of the examples it is requested to start the serial monitor to initiate the process.  That's the "while (!Serial) {;}" instruction that you will see later in the code.

Installing the extra board


The MKR1000 board is not supported by the IDE out of the box the following board should be added through the Boards Manager in the Tools menu of the IDE.

  • Arduino SAM Boards (32-bits ARM Cortex-M0+)




Installing additional libraries


While using specific sensor or some times more standard component such as the WiFi, libraries need to be added to the IDE and to project to work with them.  The good news is that when you add a new library to the IDE you often get some sample projects along with it to guide or inspire you during your development.

Adding a library can be done through the Library Manager from the Sketch menu under Include Library.  A lot of libraries can be just picked from the menu but in some cases you might need to download them from the Web and import them as well.

  • WiFi101

  • Time




Arduino MKR1000


The reasons why I choose this board amongst all the boards Arduino offers are two:

  1. It’s offered in a bundle. Since I have no background in electronic, having a standard set of components from the beginning is a preferred way to start.

  2. This board has a WiFi shield integrated out of the box.


Why Arduino rather than Raspberry Pi?  I didn't ask myself too many questions when I started this journey.  I heard about Arduino, looked a bit what were the capabilities and just went for it.

If I have to highlight the difference between the two, Raspberry Pi is a full computer with a Linux OS and all the traditional connectivity features and Arduino is much lighter, probably less expensive and maybe more appropriate to projects with very little complexity.  Why taking a bazooka to kill a fly?

Upgrade the WiFi Shield firmware


TIPS: The WiFi shield installed on the board supports:

  • WEP with HEX key

  • WAP TKIP

  • WAP2 CCMP (AES-based)


As a start, run the script provided with the library to check the firmware and upgrade it if required.  for more information on this operation, check the references mentioned at the end of this section.

Create the electronic circuit


Requirements


For those who would like to reproduce this exercise on their own, here is the bill of material with the prices at the time of writing this blog:

  • Arduino MKR IoT Bundle (84,97€ including tax and shipment fees to Belgium from the Arduino Store)

    • Arduino board MKR1000 (with headers mounted) (included in the kit)

    • Temperature Sensor TMP36 (included in the kit) or even better the DHT22 (2,08 € from aliexpress) which can also measure the humidity. At the end of the day any sensor could do the trick: light, distance, … or even simpler the WiFi signal strength for which you wouldn’t even need anything else than the board itself.

    • A couple of jump wires (included in the kit)

    • A breadboard (included in the kit)

    • Optionally, to have a visual indicator on the hardware side, to see where the process is busy (In our implementation, yellow indicates that we are in the setup method and Green in the loop excluding the ending delay)

      • Two LEDs (included in the kit)

      • Two Resistor 220 Ω (included in the kit)






Assembling the components together




Here is a detailed close up of the Arduino board to see which pins are actually used.



Schematic made using Fritzing.  The board schematic itself can be downloaded from the Arduino web site itself.

IMPORTANT NOTE: While putting the TEMP36 temperature sensor on the breadboard if you look at it with the pins up right and the flat part above, the ground is on the third pin (starting to count from the left) and 5V on the first.  If you put it the other way around as I did the sensor will heat a lot a produce a "nice" burning smell.  You will have cooked it!  Just in case.


Making the script


A couple of challenges along the road:

  • Connecting the device to the WiFi is of course supported but not with all the security system/encryption type available. Here are some restrictions:

    • WEP with HEX key is supported

    • WEP with ASCII key is not supported

    • WAP TKIP is supported

    • WAP2 CCMP (AES-based) is supported




It is not easy to know what is supported and what is not.  The easiest way to get it to work is to use tethering from your iPhone (which runs a supported WPA TKIP)…




  • Finding the best way to deal with the NTP Server.  Multiple implementations can be found on the net.

  • Finding the right way to build the HTTP requests to send the data to SCP.


Complete the script


There is multiple ways to improve the script bellow such as establishing automatically a new connection to the WiFi when the connection get lost, make the time between two cycles adjustable using a potentiometer , using other technics to keep track of time, ...  The idea here is not to make a perfect script but rather to illustrate what can be done and how easily it can be done.
#include <TimeLib.h>



#include <SPI.h>

#include <WiFi101.h>

#include <WiFiClient.h>

#include <WiFiSSLClient.h>

#include <WiFiUdp.h>



// WiFi Network specific configuration

#include "arduino_secrets.h" 

///////please enter your sensitive data in the Secret tab/arduino_secrets.h

char ssid[] = SECRET_SSID;        // your network SSID (name)

char pass[] = SECRET_PASS;        // your network password (use for WPA, or use as key for WEP)

int status = WL_IDLE_STATUS;      // the WiFi radio's status



// LEDPIN configuration

const int ledpin = 6;

const int setuppin = 1;

const int looppin = 0;

const int temperaturePin = 0;



// UDP/NTP specific configuration

unsigned int localUDPPort = 2390;         // local port to listen for UDP packets

IPAddress timeServer(129, 6, 15, 28);     // time.nist.gov NTP server

const int timeZone = 0;                   // UTC time (epoch)

const int NTP_PACKET_SIZE = 48;           // NTP time stamp is in the first 48 bytes of the message

byte packetBuffer[ NTP_PACKET_SIZE];      //buffer to hold incoming and outgoing packets

WiFiUDP Udp;                              // A UDP instance to let us send and receive packets over UDP

int statusUDP = 0;      // the WiFi radio's status



// SAP HCP specific configuration

char* host              = "iotmmsi063866trial.hanatrial.ondemand.com";

int httpsPort           = 443;

String device_id        = "63c3ab7c-5cae-4611-b6ca-d9783db25cee";

String message_type_id  = "d6cdc3d436b9859af722";

String oauth_token      = "2b8358a867bdb7678b63c1973891b95";

String url              = "/com.sap.iotservices.mms/v1/api/http/data/" + device_id;

WiFiSSLClient client;





void setup() {

  //Initialize serial and wait for port to open:

  Serial.begin(9600);

  pinMode(ledpin, OUTPUT);

  pinMode(setuppin, OUTPUT);

  pinMode(looppin, OUTPUT);

  

  while (!Serial) {

    ; // wait for serial port to connect. Needed for native USB port only

  }

  

  digitalWrite(setuppin,HIGH);

  

  // attempt to connect to WiFi network:

  Serial.print("** Connection to the WiFi network in progress: ");

  while ( status != WL_CONNECTED) {

    digitalWrite(ledpin, LOW);

    Serial.print(".");

    status = WiFi.begin(ssid, pass);      // Connect to WPA/WPA2 network:

    delay(2000);                          // wait 2 seconds for connection

  }

  Serial.println("");



  // Connected succeeded

  Serial.print("** Connection established: ");

  printIPAddress();

  digitalWrite(ledpin, HIGH);



  //NTP

  Serial.print("** Connecting to the NTP server: ");

  while ( statusUDP == 0) {

    Serial.print(".");

    statusUDP = Udp.begin(localUDPPort);

    delay(2000);  

  }

  Serial.println("");

  setSyncProvider(getNtpTime);



  //Check the connection to the Wifi Client

  Serial.println("** Connecting to the SCP MMS Service");

  if (client.connect(host, httpsPort)) {

    Serial.println("\t** Connection to SCP established");

  } else {

    Serial.println("\t** Connection to SCP failed");

  }

  

  digitalWrite(setuppin,LOW);

  Serial.println("** Setup complete !");

}



void loop() {

  digitalWrite(looppin,HIGH);

  Serial.println("\n#######################################################");

  Serial.println("#######################################################");



  //Get epoch time from the NTP Server

  time_t epoch;

  if (timeStatus() != timeNotSet) epoch = now();



  //Get the Temp from the sensor

  int sensorVal = analogRead(temperaturePin);

  float volt = (sensorVal/1024.0) * 5.0;

  float temp =(volt -.5) * 100;



// Interacting with the SCP Server

  // Check client availability and restart it if necessary

  checkClientStatus(host, httpsPort);

  

  // Send the data to SCP IoT Service

  updateSCP(epoch,temp);



  // If there are incoming bytes available from the server, read them and print them:

  getSCPResponse();

  

  digitalWrite(looppin,LOW);

  // Wait 10 seconds before looping:

  delay(7000);

}





void printIPAddress() {

  // print your WiFi shield's IP address:

  IPAddress ip = WiFi.localIP();

  Serial.print("IP Address: ");

  Serial.println(ip);

}





void updateSCP(time_t epoch, float temp) {

  String payload;

  String request;

  

  if (client.connected()) {

    payload = "{\"mode\":\"sync\",\"messageType\":\"d6cdc3d436b9859af722\",\"messages\":[{\"timestamp\":" + String(epoch) + ",\"celsiusTemp\":" + String(temp) + "}]}";

    request = "POST " + url + " HTTP/1.1\n" +

              "Host: " + String(host) + ":" + String(httpsPort) + "\n" +

              "Content-Type:application/json;charset=utf-8\n" +

              "Authorization: Bearer " + oauth_token + "\n" +

              "Content-Length: " +

              String(payload.length()) +

              "\n\n" +

              payload +

              "\n\n";

    

    Serial.println("\n## BEGIN CLIENT REQUEST ################################");

    Serial.print(request);

    Serial.println("## END CLIENT REQUEST ###################################\n");

    

    client.print(request);

  }

}



void checkClientStatus(char* host, int httpsPort) {

  if (!client.connected()){

    Serial.println("** Client restart required");

    client.stop();

    if (client.connect(host, httpsPort)) {

      Serial.println("** Client restarted");

    } else {

      Serial.println("** Client failed to start");

    }

  }

}



void getSCPResponse() {

  String response = "";

  if (client.available()) {

    response = response + "\n## BEGIN SERVER REPLY #################################\n";

    while (client.available()) {

      char c = client.read();

      response = response + c;

    }

    response = response + "## END SERVER REPLY ###################################\n";

  }

  Serial.println(response);

}





// send an NTP request to the time server at the given address

unsigned long sendNTPpacket(IPAddress& address) {

  //Serial.println("1");

  memset(packetBuffer, 0, NTP_PACKET_SIZE);   // set all bytes in the buffer to 0

  // Initialize values needed to form NTP request (see URL above for details on the packets)

  //Serial.println("2");

  packetBuffer[0] = 0b11100011;   // LI, Version, Mode

  packetBuffer[1] = 0;            // Stratum, or type of clock

  packetBuffer[2] = 6;            // Polling Interval

  packetBuffer[3] = 0xEC;         // Peer Clock Precision

  // 8 bytes of zero for Root Delay & Root Dispersion

  packetBuffer[12]  = 49;

  packetBuffer[13]  = 0x4E;

  packetBuffer[14]  = 49;

  packetBuffer[15]  = 52;



  //Serial.println("3");



  // all NTP fields have been given values, now you can send a packet requesting a timestamp:

  Udp.beginPacket(address, 123);  //NTP requests are to port 123

  //Serial.println("4");

  Udp.write(packetBuffer, NTP_PACKET_SIZE);

  //Serial.println("5");

  Udp.endPacket();

  //Serial.println("6");

}



time_t getNtpTime() {

  while (Udp.parsePacket() > 0) ; // discard any previously received packets

  //Serial.println("Transmit NTP Request");

  sendNTPpacket(timeServer);

  uint32_t beginWait = millis();

  while (millis() - beginWait < 1500) {

    int size = Udp.parsePacket();

    if (size >= NTP_PACKET_SIZE) {

      //Serial.println("Receive NTP Response");

      Udp.read(packetBuffer, NTP_PACKET_SIZE);  // read packet into the buffer

      unsigned long secsSince1900;

      // convert four bytes starting at location 40 to a long integer

      secsSince1900 =  (unsigned long)packetBuffer[40] << 24;

      secsSince1900 |= (unsigned long)packetBuffer[41] << 16;

      secsSince1900 |= (unsigned long)packetBuffer[42] << 8;

      secsSince1900 |= (unsigned long)packetBuffer[43];

      return secsSince1900 - 2208988800UL + (timeZone * 3600);

    }

  }

  //Serial.println("No NTP Response :-(");

  return 0; // return 0 if unable to get the time

}



void printDigits(int digits) {

  // utility for digital clock display: prints preceding colon and leading 0

  //Serial.print(":");

  if(digits < 10)

    Serial.print('0');

  Serial.print(digits);

}

NOTE: Some of the methods above are just taken as they are from examples provided out of the box in the IDE or the libraries...

Running the script


As you might have seen from the script, the first couple of lines are dedicated to the initialization of the connections:
** Connection to the WiFi network in progress: .

** Connection established: IP Address: 172.20.10.11

** Connecting to the NTP server: .

** Connecting to the SCP MMS Service

	** Connection to SCP established

** Setup complete !

Here is a client request as it should be formed:
## BEGIN CLIENT REQUEST ################################

POST /com.sap.iotservices.mms/v1/api/http/data/63c3ab7c-5cae-4611-b6ca-d9783db25cee HTTP/1.1

Host: iotmmsi063866trial.hanatrial.ondemand.com:443

Content-Type:application/json;charset=utf-8

Authorization: Bearer 2b8358a867bdb7678b63c1973891b95

Content-Length: 110



{"mode":"sync","messageType":"d6cdc3d436b9859af722","messages":[{"timestamp":1508752380,"celsiusTemp":69.14}]}



## END CLIENT REQUEST ###################################

The key data from this request are

  • the data endpoint with the device ID

  • the host where the sub-account can be found

  • the security token

  • the message type in the request body

  • and finally the actual data that are sent


The reply to that request should be:
## BEGIN SERVER REPLY #################################

HTTP/1.1 200 OK

Content-Type: application/json;charset=UTF-8

Transfer-Encoding: chunked

Date: Mon, 23 Oct 2017 09:52:54 GMT

Server: SAP

Strict-Transport-Security: max-age=31536000; includeSubDomains; preload



52

{"msg":"1 message(s) received from device [63c3ab7c-5cae-4611-b6ca-d9783db25cee]"}

0



## END SERVER REPLY ###################################

References




Monitoring the data coming into the SAP Cloud Platform




NOTE: As you can see the temperature values indicated on this chart are quite strange (between 50 and 100 C°) that’s because I put my sensor upside down the first time (mixing up anode and cathode) and thus cooked it…  So the values are all wrong unfortunately but it doesn't really matter for what we are trying to achieve here...

 

 



Enhancement & Troubleshooting


During my tests, I decided to change the sensor (from TMP36 to DHT22) on the board to get different and more accurate data including humidity. There is obviously a lot of other sensors that could be added or combined to make the exercise more fun.

But beside having a more elaborated construct, what is interesting in trying this is to see what is the impact on the setup when you need to handle such a change request.

Changes on the device


If you want to use the DHT22 sensor instead of the TMP36 one I would recommend to follow these steps:

  1. In Arduino IDE, import using the “Library Manager” the “Adafruit Unified Sensor” library.

  2. Import the DHT library using the import “Add .ZIP library” feature from this link: http://learn.adafruit.com/dht/downloads

  3. Then update your circuit and get your inspiration for the coding part from the example shipped with the library. Thanks Lady Ada!


Different implementations or usages of the sensor can be found on the net but this one does work.
Reference


Changes in SCP


In SCP, this operation requires a change of the message type in the IoT service.

  1. Create a new message type with an additional field

  2. Create a new device type which relies on the new message type

  3. Create a new device of the new device type


This has the following consequences on the code:

  • New Message Type ID

  • New Device ID

  • New OAuth Access Token for the device


On the IoT service side, since we have a new message type, we end up with a new table to host our measurements, T_IOT_<Message_Type_ID>, that gets created automatically upon the first data received.

Then I end up with my first real error during my implementation...

Symptoms:

I send my message and get as a response from the server a 202 back.  This means that the message got successfully delivered but couldn’t be processed yet.  In IoT, the table supposed to host the data does not get created.  I don’t get any indication on what is happening from the HTTP response I get from the server.

Where to look:

In the different tables of the service, the table T_IOT_ACKSTORE turned out to contain information that could highlight from where the problem was coming from.
































Column Value
C_DEVICEID 33d8aa08-a8a6-466a-b6c6-13c45a2795dd
C_SEQUENCEID 62dc579b-a79e-4fee-8f96-3c4f198a655e
C_STATUS VALIDATION_FAILED
C_PHASE 3
C_MSG Message type [1097e807c808684201e8] is invalid for device [33d8aa08-a8a6-466a-b6c6-13c45a2795dd]
G_CREATED Wed Nov 01 2017 10:31:01 GMT+0100 (W. Europe Standard Time)


The error here became obvious.  The device was not defined using the expected message ID.  The wrong device type was picked up from the drop down list.

Solution:

  1. Delete the device

  2. Create a new device

  3. Be sure to select the appropriate device type (since this is where the problem came from)

  4. Copy the OAuth Access Token for the new device along with the new Device ID into the script

  5. Adjust the code with the new OAuth token and device ID


 

 



What next?


We have seen how to enable the IoT service on SCP, how to configure the SCP IoT service to receive information from a device, how to assemble a simple and small device using an Arduino board that can be programmed to monitor temperature and send its measurement to SCP IoT service.

Try it yourself, it's really fun to do!

The next step is quite logic, how to consume these information from SCP.  There is naturally many scenario that we can think of but that might be a for a following post!

 

 

 

 
5 Comments
Labels in this area
Popular Blog Posts