Skip to Content
Technical Articles
Author's profile photo Niladri Podder

Real-time Object Detection with Drone and SAP Leonardo IoT

Introduction

Unmanned Aerial Vehicles (UAVs), or drones, are aircraft that can be controlled remotely by a pilot or can fly autonomously based on pre-programmed plans or automation systems. Many industries and users are adopting this technology, including military, government, commercial, and recreational users.
As a hobbyist drone builder, I have always preferred to send useful data captured from drone video to the cloud system. After successfully integrating with AWS and GCP, it was a point to associate it with SAP Leonardo IoT. The new so-called FPV (first person view) technology gave us a unique experience of flying and also able to hook up to the on-premise system which can process the video stream.

Use-case

Using Drone, identifying objects in real-time, processing the data and sending it to SAP Leonardo IoT.

Challenges

The main challenges are:

  • Deploy an object detection model on a drone micro-controller.
  • Less computation power.
  • Faster scanning systems, lower the accuracy of the objects identified in the video stream.
  • Less space to set up additional hardware, which can impact flight controls.

From the below image of the setup, believe me, we can identify the available space.

Solution

  1. FPV Technology — Using the FPV camera, we can stream the video feed and process it in Raspberry Pi Zero, which serves as an edge gateway.
  2. Object Detection — developed a React web app that takes as input from FPV camera and sends its frames to a pre-trained COCO SSD model to detect objects on it.

Components Used

  1. Drone
  2. FPV Camera
  3. Raspberry Pi Zero
  4. SAP Leonardo IoT

Flow diagram

Image: 1

Let’s Glue them.

Before I describe the complete PoC, there are two methods.

Approach 1

Stream Video feed from an FPV camera to a near edge gateway and send the transformed data to the SAP Leonardo IoT.

 

Image: 2

Approach 2

Attach RPi Zero on Drone and stream the JSON Payload to SAP Leonardo IoT. I have adhered to this method to test extensively and broaden the use-case for future implementations.

  1. Drone — Nowadays cheap drones are available, which can be used indoors. Anyone who wants to go with approach 1 (Image:2) can use small drones.  As I have followed approach 2 (Image:1), I have used a customized frame for building the drone and attached additional parts to it. There are lots of tutorials available to build a drone from scratch.
  2. FPV camera — FPV, or First Person View, is a concept in the drone hobby where a live video stream from the drone is sent to a display in the pilot’s view. There are two popular FPV varieties:
    • 2.4GHz digital (WiFi) — Widely used in small drones and works with any Android/iOS smartphone (or tablet) on the market.
    • 5.8GHz — 5.8GHz is the preferred frequency for hobbyists and professionals alike. This is because the 5.8GHz sits nicely between the high range and high bandwidth, which means it is the most versatile of them all. As a general rule, the higher the frequency, the lower the range. In other words, higher frequency = less range, clearer video. That said, people from across the world have reported miles of range with 5.8GHz FPV.
  3. Raspberry Pi Zero — Raspbian OS installed with Node.js. Flask API which can connect to SAP Leonardo IoT using MQTT protocol.
  4. SAP Leonardo IoT Service — Onboarded a device as Drone and added Property as objects and Value as the identified object’s name.

How it works!

For this use-case, I have used Deep Learning as this is one of the only methods by which we can overcome the challenges of feature extraction. Basically, deep learning mimics the way our brain functions i.e. it learns from experience.

The model which I have used for object detection is COCO-SSD (Common Objects in Context – Single Shot Detector). Single Shot Detector is a neural network architecture that uses a “proposal generator,” a component whose purpose is to search for regions of interest within an image. After the ROI (Region of Interest) has been identified, feature extraction is used to get the features of the region and determine the objects identified. COCO dataset has images that are one of the 90 classes, such as cell-phone, mouse, person, bus, cycle, kite and so on.

I have registered a device with the name “Drone” in SAP IoT Cockpit to get the list of objects identified from the video stream. As the number of objects identified is large, so I have used a rule to only send data when the object “Cup” is detected.

SAP Leonardo IoT Service Cockpit view: 

 

Here, what my Drone can see!

As I am flying the drone indoor, it is set to 5 feet as maximum altitude from the ground level.

Conclusion

With the above setup, it will be really helpful to deploy drones for the rescue operations and maintain a data log of the objects which are identified in SAP Leonardo IoT services. Further, we can add multiple components to extend its purpose, like feature extraction, image analytics and so on.

 

Assigned Tags

      5 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Umberto Panico
      Umberto Panico

      Great.

      Which alghiritm to recognize objects in video?

      Author's profile photo Niladri Podder
      Niladri Podder
      Blog Post Author

      Hi Umberto,

      I have used SSD(Single Shot Multibox Detector) Algorithm.

      Author's profile photo Yasser SABER
      Yasser SABER

      Hi Niladri,

      This is Yasser from the KSA Presales lab.

      Really, I like what you are done with drone and object detection.

      My ask, about if you have step by step details on how we can achieve this task.

      We already built the object detection part with raspberry pi , how we can connect it with Leonardo , MQTT part and especially this part “a rule to only send data when the object “Cup” is detected”

      Author's profile photo Mohamed Judi
      Mohamed Judi

      Thank you, Niladri, for sharing this valuable knowledge. I have a customer interested in using drones to recognize anomalies and degradation in concrete structures as part of a preventive maintenance effort. Do you have examples or any information on how we can use SAP Intelligent Technologies such as IoT and ML to build a POC?

      Author's profile photo Niladri Podder
      Niladri Podder
      Blog Post Author

      Hi Mohamed,

      Earlier I have worked on identifying cracks under Bridges, for which I have collected all the Crack images and retrain with SDNET2018 (Concrete crack detection image dataset). Using the same method, this model can be deployed on Drone to provide real-time outlier detection. 

      Thanks,

      Niladri