Real-time Object Detection with Drone and SAP Leonardo IoT

Introduction

Use-case

Using Drone, identifying objects in real-time, processing the data and sending it to SAP Leonardo IoT.

Challenges

The main challenges are:

• Deploy an object detection model on a drone micro-controller.
• Less computation power.
• Faster scanning systems, lower the accuracy of the objects identified in the video stream.
• Less space to set up additional hardware, which can impact flight controls.

From the below image of the setup, believe me, we can identify the available space.

Solution

1. FPV Technology — Using the FPV camera, we can stream the video feed and process it in Raspberry Pi Zero, which serves as an edge gateway.
2. Object Detection — developed a React web app that takes as input from FPV camera and sends its frames to a pre-trained COCO SSD model to detect objects on it.

Components Used

1. Drone
2. FPV Camera
3. Raspberry Pi Zero
4. SAP Leonardo IoT

Flow diagram

Image: 1

Let’s Glue them.

Before I describe the complete PoC, there are two methods.

Approach 1

Stream Video feed from an FPV camera to a near edge gateway and send the transformed data to the SAP Leonardo IoT.

Image: 2

Approach 2

Attach RPi Zero on Drone and stream the JSON Payload to SAP Leonardo IoT. I have adhered to this method to test extensively and broaden the use-case for future implementations.

1. Drone — Nowadays cheap drones are available, which can be used indoors. Anyone who wants to go with approach 1 (Image:2) can use small drones.  As I have followed approach 2 (Image:1), I have used a customized frame for building the drone and attached additional parts to it. There are lots of tutorials available to build a drone from scratch.
2. FPV camera — FPV, or First Person View, is a concept in the drone hobby where a live video stream from the drone is sent to a display in the pilot’s view. There are two popular FPV varieties:
   • 2.4GHz digital (WiFi) — Widely used in small drones and works with any Android/iOS smartphone (or tablet) on the market.
   • 5.8GHz — 5.8GHz is the preferred frequency for hobbyists and professionals alike. This is because the 5.8GHz sits nicely between the high range and high bandwidth, which means it is the most versatile of them all. As a general rule, the higher the frequency, the lower the range. In other words, higher frequency = less range, clearer video. That said, people from across the world have reported miles of range with 5.8GHz FPV.
3. Raspberry Pi Zero — Raspbian OS installed with Node.js. Flask API which can connect to SAP Leonardo IoT using MQTT protocol.
4. SAP Leonardo IoT Service — Onboarded a device as Drone and added Property as objects and Value as the identified object’s name.

How it works!

For this use-case, I have used Deep Learning as this is one of the only methods by which we can overcome the challenges of feature extraction. Basically, deep learning mimics the way our brain functions i.e. it learns from experience.

The model which I have used for object detection is COCO-SSD (Common Objects in Context – Single Shot Detector). Single Shot Detector is a neural network architecture that uses a “proposal generator,” a component whose purpose is to search for regions of interest within an image. After the ROI (Region of Interest) has been identified, feature extraction is used to get the features of the region and determine the objects identified. COCO dataset has images that are one of the 90 classes, such as cell-phone, mouse, person, bus, cycle, kite and so on.

I have registered a device with the name “Drone” in SAP IoT Cockpit to get the list of objects identified from the video stream. As the number of objects identified is large, so I have used a rule to only send data when the object “Cup” is detected.

SAP Leonardo IoT Service Cockpit view: 

Here, what my Drone can see!

As I am flying the drone indoor, it is set to 5 feet as maximum altitude from the ground level.

Conclusion

With the above setup, it will be really helpful to deploy drones for the rescue operations and maintain a data log of the objects which are identified in SAP Leonardo IoT services. Further, we can add multiple components to extend its purpose, like feature extraction, image analytics and so on.