In this article I will try to cover the concept of using the Augmented Reality with SAP MII to show the information of plant on the mobile device and execute the necessary actions. This article will cover the end to end scenario of using the existing application and building the mobile based augmented reality application. This demonstration uses geo-location of plant machinery (Boilers & Production Line) to recognize them individually. The same can also be achieved using the marker image for recognition. After recognizing the machinery, various mapped actions can be executed. For demonstration we build the temperature history report or user can call supervisor action.
Software needed to demonstrate the scenarios are:
- Layar Reality Platform (Layar Server hosted on web)
- Layar iPhone SDK (iOS SDK to develop the mobile app on top of it)
- PHP / Servlets (Custom services exposed to layar server for providing the information).
- SAP MII.
SAP MII
SAP MII is Manufacturing Integration and Intelligence. It helps to integrate the shop floor data to enterprise server. This helps manufacturing plants to be adaptive with the current operations. It gives real time access to the shop floor data such as orders, materials, equipment status, costs and product quality. MII provides you the platform to create your manufacturing processes and link it with master data to get a single version of the truth. Plant Manager / Engineer is empowered to take decisions to improve the performance of the plant operation.
- Manufacturing integration – SAP MII seamlessly integrates plant processes with business operations. The application provides a single, standards-compliant connection between shop-floor activities and enterprise systems for ERP, manufacturing execution, and sales-force automation.
- Manufacturing intelligence – The real-time analytics engine in SAP MII aggregates, calculates, and delivers a single view of relevant events, alerts, key performance indicators (KPIs), and decision support. Production personnel access all relevant information through a single, role-based dashboard. Empowering them to make fast, accurate decisions that can improve your bottom line.
Augmented Reality
Augmented Reality is a mix of real world to virtual world. Good examples of augmented Reality can be seen in the popular movie Terminator, in which Arnold’s eye was augmented with the information of the objects or people while looking at it. This enables him to take further actions.
Augmented Reality Demonstration with MII
The demonstration creates the Manufacturing Plant scenario. The plant has boilers and production lines. For a Boiler or Production Line user wants to get the information such as temperature, status, pressure report etc.. while pointing the mobile at machinery(Boilers/Production Lines) and take appropriate actions. Actions such as ‘Call Supervisor’, ‘Show Temperature History’, ‘Take me to Boiler’, ‘Show Status’ etc…
As we know that the machinery in the manufacturing plants is fixed so they can be easily mapped with the geo-locations. In our demo we used the geo-location of objects for recognizing the Boiler or Production Lines.
Diagram for the overall Software Components of the Solution/Demo:
Each Boiler is mapped at specific place as shown in the MAP View below.
Layar Server Platform:
Layar Platform is used for augmented reality implementation. In the Layar Platform allow developers to configure the layer/application. In the demo MII demo is configured to interact with MII Augmented Server, this is done by specifying the URL of MII Augmented Server. Layar also provides the SDK for iPhone, which was used to build independent augmented reality browser based mobile application.
Layar.com is the platform for the augmented reality browsing, in this platform applications are configured as layers. For our demonstration we created the MII Layer. In the configuration screen the main parameter is the URL of the MII Augmented Server Application which will return the information of POI’s (Point of Interest) such as boilers and production lines.
MII Augmented Server:
This is the web-based application server which gets called by Layar Platform Server to get the POI’s and its specific actions. Local Database is created to store the information of Boiler mapping to geo-positioning.
MII Augmented Server implemented the logic to extract the following information from its database:
- Information of all POI’s near the current location. For each boiler/production line locations are mapped as:
- 1, Boiler1, 28.122222, 77.343434,
- 2, Boiler2, 28.122223, 77.343434,
- 3, Boiler3, 28.122224, 77.343434,
Etc..
(see the difference in the 6th decimal place which might shows the distance of 100meters).
- Stores the information of ‘Actions’ of all POI. For each boiler/production line action database is created as:
- Boiler1, Call Action, Call Supervisor, +91 98N NNN NNNN
- Boilder1, Info, Temperature History, http://yourserverURL /historyreport1
- Boiler2, Call Action, Call Supervisor, +91 98N NNN NNNN
- Boilder2, Info, Temperature History, http://yourserverURL /historyreport2
Etc..
- Action can be of type Call, Info, Video & Audio etc. content is accordingly stored in the database.
MII Augmented Server created the database tables for storing information such as geo-position (latitude & longitude), status, temperature, reports, pressure etc.. and also actions of each POI (boiler & production). This is consumed by the Layar platform and Layar mobile SDK.
MII Mobile App:
Layar provides two ways to expose your content to the mobile user.
Layer in reality browser:
The simplest of the way is to add it as the layer inside the Layar reality browser. In this way the end user will install the Layar Reality Browser. Inside the browser user can reach to your published content via search of your layer name.
Layar Player SDK:
Another approach to publishing your content is by using the Layar SDK and creating an independent mobile application for iPhone and end user just need to download your application and has no dependency with Layar Reality Browser.
In our case we used the Layer Player SDK and created the MII Mobile application for user to easily launch and use it.
Above picture shows, how the communication between each software layer happens for fetching the point of interests with current location of mobile GPS. Layar Server fetches the URL from the Layar configuration and calls the URL (MII Augmented Server) along with the geo-position & parameters in query string. MII Augmented Server uses the parameters and geo-locations to send back the details of Point Of Interest for the end users. Layar Server further sends this information to the Mobile device (Layar SDK) to show the information to the end user. End user gets the information of the objects near him and user can select any POI and to know its various actions.
Once the end user views all the POI’s near its location, user can select Boiler1 and display shows all the Action mapped for Boiler1. User seeks for the temperature history report for Boiler1. Upon invoking this action call goes to MII Augmented Server to return the temperature report in HTML format to mobile. MII Augmented server in turn calls the SAP MII Service to fetch the temperature historical report.
Information of boiler & production line can be viewed in multiple ways. Layar Player SDK support three different views :
1. Camera mode: In this mode mobile camera is running and end user can see the information of object. This is an interesting view since it displays the information of the object to which your mobile is pointing to. Top right shows the radar and the current selected object. This view uses the mobile GPS & accelerometer to understand direction the mobile is pointing at and appropriately adjusting the display of machine.
As shown the image below:
2. List View: In this mode the typical list view is created on mobile displaying all the boilers near your location as a list.
3. Map View: In this view google map is used to display each boiler.
Image showing the Actions on Boiler2:
Above image shows the mapped action with Boiler-2. End user can simply touch and invoke the appropriate actions to get temperature history report or call supervisor.
Layar Player SDK also provides the client side setting screen in mobile user can configure application specific parameter, such as machines (boilers or production lines) POI’s. User can also specify the search range; it is the radius for searching the POI’s around the mobile device. Device uses the GPS & accelerometer of mobile device to gets its latitude, longitude & direction.
With the idea of plant machinery being stationary in the plant, mapped with its geo-position we can extract the data regarding the machinery using the mobile based interface. Ease of consuming the information using the Augmented Reality concepts makes huge difference in using the existing applications.
Above approach can be applied in many other industries such as Retail, SCM etc.. In retail each store or store area can be mapped and the performance of availability of product can be checked using the Augmented Reality technique. In a warehouse scenario user can look into the products inside the warehouse and left over storage capacity of each bin/stack/level. Another popular scenario is in Real Estate industry, in which users get to know the apartment availability or price while looking at it. This is a great mix of technology with the need of end user to mix the virtual information with real world objects. In a big event spread in a wide area such as SAP TechEd, same technique can be used for showing participants various tracks or presentation. This would allow participant to have brief or detailed information or video of the track or session. It has good potential to be applied in various scenarios.
Main challenge however remains with business application for a company is to have their own augmented reality server, instead of depending upon third party server such as Layar and provide end to end solutions with support.
I would like to thank Gaurav who worked with me to create and materialize the whole concept and implemented such a beautiful iPAD demo. Enjoy watching the demo here.