Skip to Content

LeafSnapThere’s  a new app for the iPhone, called LeafSnap. With the app, you can take a  photo of a leaf, and it will tell you what type of tree it belongs (or  belonged) to.

It uses the iPhone camera and a bit of smart logic. All you need to do is collect a leaf, place it against a white background, and take a photo of it. The application takes care of the rest – it will extract the shape of the leaf and then compare it against a database of known (currently North American) trees.

As such, the recognition is not yet perfect, you will be presented with a shortlist of leaves/trees to consider. But still, within seconds you will be able to find the right one and stop wondering (or arguing with your partner) what the name of the tree could be.

Pretty  smart, isn’t it? Can you think of other situations, where similar  technology would be useful? How about trying to tell whether that spider  in your bathroom is venomous or not? Or maybe you want to find out  whether the glass you were given is for white wine or for water?

But  you may just as well ask why should you care? Why am I writing about it here? Well, that’s exactly the point. Let me try to explain how relevant to SAP these  technologies may be.

Can  you think about an enterprise application, which requires you to  connect a webcam to your computer and point it at something in order to  perform some meaningful action? How about an inventory search  application which can glimpse at that screw you’re holding in your hand  and show you the stock levels and the location of a box, which contains  more such screws, in the warehouse?

Or,  can you think about a mobile application that lets you “look” at a  30 foot shipping container, shows the contents of the container (augmented reality), and, let’s say –  ports of origin and destination of that container? How cool would it be to inspect a container ship with an app that could do that?

I haven’t seen anything like that. At least not yet. I am still waiting for image processing and augmented reality technologies to become widely used in the enterprise space.

Those of you who saw some of the things I and my team work on, know  that we’re not really just sitting and waiting, but we do try to shape  that space a bit.

 

Just as a disclaimer, I am aware of fantastic work done at SAP Research in the space of augmented reality. SiWear is an excellent example of industry relevant augmented reality application. However, what I am focusing on in this blog post is applications of augmented reality using consumer electronics, and specifically smartphones that each of us owns (or soon will). Therefore, I won’t focus much on SiWear, but will skip straight to two projects that focus on using readily available hardware.

 

In this blog post I want to write about two applications – ARNIE and Holodeck – research prototypes that we have been working on to try to promote the idea of image/video processing in enterprise applications. Both of them are a result of a bigger  teamwork, and I am lucky enough to be able to have the best people ever  in these teams. These projects are exploring the ability to  analyse the world as viewed by a computer camera, and do something  meaningful with it. The first one (ARNIE) could be described as a barcode reader  with an augmented reality and enterprise twist. The second one (Holodeck),  something we still work on, is exploring the area of image processing even further and potentially  showing a completely new direction for such applications.

Project 1: ARNIE

Some  of you might have seen ARNIE at TechEd 2010 DemoJam in Berlin. Because  of an epic fail (we weren’t able to properly share the screen using a VGA cable) during  the demo, I don’t particularly like to go back to the recording of that  demo. But we still made it to the 3rd place (out of ca. 200 submissions), which means that at least  some of the TechEd attendees liked it.

So  let me share another video of it, one that shows exactly the same  functionality we demonstrated at TechEd, but in a bit different  environment.

In ARNIE, we wanted to focus on three following things:

1.  Augmented Reality, but one that doesn’t use GPS and its friends  (compass, accelerometer, gyroscope), and instead focuses only on  analysis of the viewed image. Therefore, it works well outdoors as well as indoors.

2. Backend integration: whatever we see, is a pointer to an object in an enterprise system. We have built a RESTful server application that is a hub for all ARNIE clients. It is able to forward requests to respective enterprise systems, and the results back to ARNIE clients. Thanks to that approach, we are not concerned with the mobile phone’s ability to talk to an enterprise backend. Think about possibilities that open up when the ARNIE backend links to our Sybase Unwired Platform!

3. Extensibility: the application is written once and can be configured to behave differently based on the QR code it sees. Already in the video you can see three types of reactions to QR codes: (1) retrieving document status and showing that as an icon, (2) retrieving calendar entries and showing them as text, (3) retrieving graphical files and displaying them.

ARNIE, in a sense is a fairly simple application. But we really see two differentiators in this piece of software. The first one is augmented reality – forget about barcode readers. You can take ARNIE and look at a stack of documents on your desk – it will process all of them at the same time and display status icons in the right spots. The second one is coming back to the paradigm that QR code is only identifying the object. We don’t like the idea of storing all information in a QR code. We just treat it as a pointer to the real-time, up-to-date information about an object in an enterprise system.

If the whole approach makes sense to you, feel free to reach out to me to talk about ARNIE!

Project 2: Holodeck

In Holodeck we are trying to address a problem of collaboration across locations.

Imagine a situation where you start a brainstorming session in your office and later on you want to involve colleagues from another office, possibly located in another country. You probably start in front of a whiteboard, as this seems to be the most efficient way of brainstorming. And later on…

image

Exactly, what happens later on? Do you take a photo of the whiteboard and send it to your remote colleagues? Do you use a webcam to share the view of the whiteboard with others? Or maybe you switch to an online collaboration tool to enable the remote colleagues to contribute to the discussion in the same way you do it? Each of these approaches has its shortcomings. The photo is out-of-date the very moment someone makes a change to the content on the whiteboard, and I am sure you are not willing to keep taking photos everytime something changes. The webcam doesn’t give you a resolution that is good enough, and so your colleagues will likely not be able to read everything you put on the whiteboard. Both of these approaches do not let your colleagues actively modify the content of the whiteboard. The third approach (online collaboration tool) is solving the problems above, but introduces a new one: brainstorming doesn’t seem to be working well in front of a keyboard and a screen.

Oh, and there’s another option. You can spend ten thousand dollars or so, and equip both rooms in smart boards. Some of them let you share content with another location. None of them, though, is good at supporting the most basic tools: markers and post-it notes. If you prefer a very natural interaction style, smart boards may be not good for you, not even mentioning the cost and other limitations.

So, we asked ourselves: can we try to come up with a better solution?  Can we build a better solution? And that’s how the idea of Holodeck was born. We decided to build a software solution, that would not require any investment in hardware (assuming use of standard hardware that can be found in any office) and become a great tool for supporting brainstorming – both remote and local.

image

Enough talking about it. Have a look at this video. It shows what our prototype is capable of. Bear in mind – this video only demonstrates technology. We do not try to demonstrate any particular use case.

There is quite a bit of SAP technology behind. In particular, we make heavy use of the collaborative process modelling tool in SAP StreamWork. You might have seen the tool Gravity – Collaborative Business Process Modelling within Google Wave, and of course you are aware of SAP StreamWork.

Treat that small video as a teaser. I will wait to see your reactions in comments to this blog, and if you like it, I will post more information about the application. All I can say, is that we have progressed quite a bit since we shot that video. The iPhone application experience is similar to one of using Skype. But instead of sharing video or chatting, you share whiteboards. We are integrating with StreamWork and some of SAP’s customers and partners have already seen it. Thanks to the integration with the process modelling tools, we can support process brainstorming that seamlessly leads to process modelling and further process execution.

Thanks to the integration with StreamWork (ability to view and “replay” whiteboards even when they physically do not exist anymore) and support of very basic hardware, Holodeck can support all sorts of brainstorming activities, as well as anything else that requires a whiteboard (think: kanban, agile development planning, task assignment…).

image

Is there a use case you could think of? Do you think an application like Holodeck could help you in your work? Let me know. We are constantly working on improving the prototype before making it available for testing, and we would be more than happy to hear about your needs!

And, by the way, the next time you wander through a forest and try to tell the names of the trees around you, think about SAP and applications of image processing in enterprise software!

To report this post you need to login first.

6 Comments

You must be Logged on to comment or reply to a post.

  1. Ajay Das
    I would love to have such a tool where a teacher has an interactive whiteboard shared with a remote classroom.
    (0) 
  2. Clinton Jones
    i like these ideas though I think the part identifier has been done, just not for use with SAP and as far as shipping containers are concerned visual packing manifests have been an option for vessels and railway trains for some time but not for actual containers. An interesting concept nonetheless.
    (0) 
    1. Marek Kowalkiewicz Post author
      Thanks Clinton!
      I agree, the concepts shown in ARNIE have been applied here and there. I’d still claim that we could do better – as in, using cheap consumer electronics rather than expensive AR hardware/barcode scanners. Plus of course the SAP integration.
      Again – you’re fully right. I guess it’s more about trying to make it more pervasive…
      (0) 
  3. Chris Paine
    I’d love to be able to do a more gesture (finger painting) on a tablet approach to gravity input. would be awesome to be able to draw a line on a tablet screen to link two post-it notes that someone has put on a whiteboard.

    I could also see a use for collaborative UI design – would be great to draw UI on a whiteboard, recognise it and capture it electronically – also allowing for easy electronic capture of remote collaborations.

    Very very cool stuff! Thanks for posting it up. 

    (0) 
    1. Marek Kowalkiewicz Post author
      Thanks for your comments, Chris! At this stage we’re using the vanilla gravity client in Holodeck, but we’ve started discussing developing a native version of it. Right now drawing a line between two post-it notes, as you describe it, is possible, but you need to tap on a post it note, choose the line tool and then tap on the target post it note. I see your point in finger painting and I agree that would be a very natural approach. Let’s see if we can do it, added to our wish-list! 🙂

      Re UI design – I fully agree! The version that we have at the moment should in principle be able to do it (if you draw an enclosed shape around anything, it will be picked up by Holodeck), but we never tested such a use case…

      (0) 

Leave a Reply