It uses the iPhone camera and a bit of smart logic. All you need to do is collect a leaf, place it against a white background, and take a photo of it. The application takes care of the rest – it will extract the shape of the leaf and then compare it against a database of known (currently North American) trees.
As such, the recognition is not yet perfect, you will be presented with a shortlist of leaves/trees to consider. But still, within seconds you will be able to find the right one and stop wondering (or arguing with your partner) what the name of the tree could be.
Pretty smart, isn’t it? Can you think of other situations, where similar technology would be useful? How about trying to tell whether that spider in your bathroom is venomous or not? Or maybe you want to find out whether the glass you were given is for white wine or for water?
But you may just as well ask why should you care? Why am I writing about it here? Well, that’s exactly the point. Let me try to explain how relevant to SAP these technologies may be.
Can you think about an enterprise application, which requires you to connect a webcam to your computer and point it at something in order to perform some meaningful action? How about an inventory search application which can glimpse at that screw you’re holding in your hand and show you the stock levels and the location of a box, which contains more such screws, in the warehouse?
Or, can you think about a mobile application that lets you “look” at a 30 foot shipping container, shows the contents of the container (augmented reality), and, let’s say – ports of origin and destination of that container? How cool would it be to inspect a container ship with an app that could do that?
I haven’t seen anything like that. At least not yet. I am still waiting for image processing and augmented reality technologies to become widely used in the enterprise space.
Those of you who saw some of the things I and my team work on, know that we’re not really just sitting and waiting, but we do try to shape that space a bit.
Just as a disclaimer, I am aware of fantastic work done at SAP Research in the space of augmented reality. SiWear is an excellent example of industry relevant augmented reality application. However, what I am focusing on in this blog post is applications of augmented reality using consumer electronics, and specifically smartphones that each of us owns (or soon will). Therefore, I won’t focus much on SiWear, but will skip straight to two projects that focus on using readily available hardware.
In this blog post I want to write about two applications – ARNIE and Holodeck – research prototypes that we have been working on to try to promote the idea of image/video processing in enterprise applications. Both of them are a result of a bigger teamwork, and I am lucky enough to be able to have the best people ever in these teams. These projects are exploring the ability to analyse the world as viewed by a computer camera, and do something meaningful with it. The first one (ARNIE) could be described as a barcode reader with an augmented reality and enterprise twist. The second one (Holodeck), something we still work on, is exploring the area of image processing even further and potentially showing a completely new direction for such applications.
Project 1: ARNIE
Some of you might have seen ARNIE at TechEd 2010 DemoJam in Berlin. Because of an epic fail (we weren’t able to properly share the screen using a VGA cable) during the demo, I don’t particularly like to go back to the recording of that demo. But we still made it to the 3rd place (out of ca. 200 submissions), which means that at least some of the TechEd attendees liked it.
So let me share another video of it, one that shows exactly the same functionality we demonstrated at TechEd, but in a bit different environment.
In ARNIE, we wanted to focus on three following things:
1. Augmented Reality, but one that doesn’t use GPS and its friends (compass, accelerometer, gyroscope), and instead focuses only on analysis of the viewed image. Therefore, it works well outdoors as well as indoors.
2. Backend integration: whatever we see, is a pointer to an object in an enterprise system. We have built a RESTful server application that is a hub for all ARNIE clients. It is able to forward requests to respective enterprise systems, and the results back to ARNIE clients. Thanks to that approach, we are not concerned with the mobile phone’s ability to talk to an enterprise backend. Think about possibilities that open up when the ARNIE backend links to our Sybase Unwired Platform!
3. Extensibility: the application is written once and can be configured to behave differently based on the QR code it sees. Already in the video you can see three types of reactions to QR codes: (1) retrieving document status and showing that as an icon, (2) retrieving calendar entries and showing them as text, (3) retrieving graphical files and displaying them.
ARNIE, in a sense is a fairly simple application. But we really see two differentiators in this piece of software. The first one is augmented reality – forget about barcode readers. You can take ARNIE and look at a stack of documents on your desk – it will process all of them at the same time and display status icons in the right spots. The second one is coming back to the paradigm that QR code is only identifying the object. We don’t like the idea of storing all information in a QR code. We just treat it as a pointer to the real-time, up-to-date information about an object in an enterprise system.
If the whole approach makes sense to you, feel free to reach out to me to talk about ARNIE!
Project 2: Holodeck
In Holodeck we are trying to address a problem of collaboration across locations.
Imagine a situation where you start a brainstorming session in your office and later on you want to involve colleagues from another office, possibly located in another country. You probably start in front of a whiteboard, as this seems to be the most efficient way of brainstorming. And later on…
Exactly, what happens later on? Do you take a photo of the whiteboard and send it to your remote colleagues? Do you use a webcam to share the view of the whiteboard with others? Or maybe you switch to an online collaboration tool to enable the remote colleagues to contribute to the discussion in the same way you do it? Each of these approaches has its shortcomings. The photo is out-of-date the very moment someone makes a change to the content on the whiteboard, and I am sure you are not willing to keep taking photos everytime something changes. The webcam doesn’t give you a resolution that is good enough, and so your colleagues will likely not be able to read everything you put on the whiteboard. Both of these approaches do not let your colleagues actively modify the content of the whiteboard. The third approach (online collaboration tool) is solving the problems above, but introduces a new one: brainstorming doesn’t seem to be working well in front of a keyboard and a screen.
Oh, and there’s another option. You can spend ten thousand dollars or so, and equip both rooms in smart boards. Some of them let you share content with another location. None of them, though, is good at supporting the most basic tools: markers and post-it notes. If you prefer a very natural interaction style, smart boards may be not good for you, not even mentioning the cost and other limitations.
So, we asked ourselves: can we try to come up with a better solution? Can we build a better solution? And that’s how the idea of Holodeck was born. We decided to build a software solution, that would not require any investment in hardware (assuming use of standard hardware that can be found in any office) and become a great tool for supporting brainstorming – both remote and local.
Enough talking about it. Have a look at this video. It shows what our prototype is capable of. Bear in mind – this video only demonstrates technology. We do not try to demonstrate any particular use case.
There is quite a bit of SAP technology behind. In particular, we make heavy use of the collaborative process modelling tool in SAP StreamWork. You might have seen the tool Gravity – Collaborative Business Process Modelling within Google Wave, and of course you are aware of SAP StreamWork.
Treat that small video as a teaser. I will wait to see your reactions in comments to this blog, and if you like it, I will post more information about the application. All I can say, is that we have progressed quite a bit since we shot that video. The iPhone application experience is similar to one of using Skype. But instead of sharing video or chatting, you share whiteboards. We are integrating with StreamWork and some of SAP’s customers and partners have already seen it. Thanks to the integration with the process modelling tools, we can support process brainstorming that seamlessly leads to process modelling and further process execution.
Thanks to the integration with StreamWork (ability to view and “replay” whiteboards even when they physically do not exist anymore) and support of very basic hardware, Holodeck can support all sorts of brainstorming activities, as well as anything else that requires a whiteboard (think: kanban, agile development planning, task assignment…).
Is there a use case you could think of? Do you think an application like Holodeck could help you in your work? Let me know. We are constantly working on improving the prototype before making it available for testing, and we would be more than happy to hear about your needs!
And, by the way, the next time you wander through a forest and try to tell the names of the trees around you, think about SAP and applications of image processing in enterprise software!