Siri.Drone – The Back Story
At the 2012 Aussie Demo Jam, held as part of the SAP Inside Track Sydney and Mastering SAP Technologies event on March 25th 2012, I showcased a demo that used a Parrot AR.Drone helicopter, ABAP and the Siri voice activated assistant that comes on the latest Apple iPhone 4S.
Since then many people have asked varying questions about what was shown, how it worked, how it was built and why you would do it anyway.
This blog is a bit of a ramble about how I came to do this demo – and why it wasn’t part of the Demo Jam competition itself.
For those of you not interested in this you might be more interested in this post that describes how this demo was built. It also includes lots of video as well.
The Back Story
I find myself taking a fairly active and visible role in most things around the Mastering SAP Technologies conferences in Australia. As well as presenting some sessions I am usually asked to introduce some of the speakers, facilitate some of the discussion groups, help organise the SAP Inside Track sessions and, most recently, be the Master of Ceremonies at the Australian Demo Jam.
As The Eventful Group were preparing for the 2012 Mastering SAP Technologies conference they decided to ask one of the SAP executives who was coming to Australia to take the role of Demo Jam MC. So in late November 2011 they approached me and asked me to consider entering Aussie Demo Jam 2012 as they felt the audience would be interested in what I might come up with. I told them I would think about it, and over the next month or so I really formulated the whole idea.
My initial thoughts were that if I was going to enter Demo Jam I was bloody well going to try and win it! This might sound obvious – but I have got to tell you that the Aussies come up with the most amazing Demo Jam ideas and I had a fair idea that the usual suspects (including former winners Matt Harding, Alisdair Templeton, Thomas Jung and Chris Paine) would all be entering. This would be a tough field to beat. In fact I was pretty sure I would have to rely on the showmanship of the demo rather than the technical content to win. So I started searching for ideas that would make a big visual impact on the audience – hoping to subvert my expected potential rivals who would no doubt produce some mind-blowing technology. In other words I was going to rely on selling the sizzle rather than the steak.
I very quickly started exploring the idea of using the helicopter – or more specifically the Parrot AR.Drone. Chris Paine & Neil Gardiner had won Aussie Demo Jam 2011 with an idea that utilised the AR.Drone – unfortunately due to technical difficulties they didn’t actually get it to fly. I thought that if I could make the AR.Drone fly it would provide exactly the sort of visual experience for the audience to grab their attention – and hopefully their votes. I figured if Chris and Neil had won without even getting the drone to fly I surely had a chance if I could get mine off the ground.
I knew nothing about the AR.Drone so the first thing I did was take a look at the SDK and the associated Developers Guide. This helped me understand what the AR.Drone was capable of and what I could reasonably expect to be able to get it to do.
I discovered that you connect to the AR.Drone using WiFi and communicate to it via several UDP ports. You control and configure the AR.Drone by sending “AT commands” (remember configuring modems?) on port 5556. Information about the AR.Drone is sent to the client application on port 5554. This includes stuff like status, position, speed, etc. and is called navdata. There is also a video stream from the drone onboard cameras sent on port 5555. The SDK included a codec for this video format. I figured I could, with some effort, get an ABAP application talking to the drone. I wasn’t sure exactly how I would do that but I thought it would be possible.
I also discovered that the drone SDK included data for so called “tag detection” that is used in gaming. These “tags” are actually patterns painted on the drone shell or stuck on with stickers and are used in multi-drone scenarios. Essentially the drone can recognise other drones so you can build multiplayer dog-fight style games. This sounded great! If I could use ABAP to get the drone to target an enemy and then destroy it that would get the audience attention – especially if I destroyed it in spectacular fashion. Evil plans started to form now. They involved some pyrotechnics and a green laser pointer I was accidentally sent as a free gift from thinkgeeks.com. (*Note These green lasers are illegal in Australia) I actually approached Sophie Sipsma from Eventful about the possibility of getting a small, but loud and highly visible, explosion to ensure total destruction of my target. (Insert Dr. Evil laughter here.)
By now I am pretty sure I am onto a winner. Helicopters, lasers, pyro. It can’t fail really – except for one piece. Experience tells me that recent demo jam winners have tended to incorporate some sort of mobile device to really wow the crowd. I need a mobile solution as well. Co-incidentally with Christmas approaching I decided to buy both my wife and daughter iPhones as Christmas presents. My daughters’ phone, a Samsung I think, had just died and my wife was over her original iPhone and ready for an upgrade. As soon as my daughter got her iPhone she started playing with Siri – the voice activated personal assistant that comes with the iPhone 4s. For me the immediate thing I liked about Siri was how easily and quickly it interpreted what was said and turned it into an action. I recalled that not long previously I had seen a post and demo of a Siri proxy which intercepted Siri commands and executed custom actions. Cool. Surely even the ABAP Freak couldn’t top that!
Then, a major show stopper! The person who was asked to MC the Demo Jam event had to pull out and I was asked to step in. This created a potential conflict of interest for me as one of my roles as Demo Jam MC is to help select the finalists who will compete on stage from the many submissions. This means, should I stay in the contest, that I would be in the position of knowing what the other competitors were planning and potentially they might approach me to assist them with ideas and logistics during the preparation phase.
By this time I had gotten so far down the track that I was reluctant to drop my idea, but I did stop working on it and came up with an alternative way forward. I figured that if I handed the idea and prototype over to someone else to present that would be okay as long as none of the other entries were doing anything similar.
When the selection committee met we went though and selected a short list of finalists from the entries. I was relieved that nothing selected was in any way similar to my idea so I then proposed it to the other committee members with the provision that I would have someone else present it – I even had the person in mind. My fellow selection committee members were happy with this arrangement and ultimately my idea was selected as one of the 6 finalists chosen to present at Aussie Demo Jam.
I then approached my preferred alternate presenter and asked if they would be willing to step in and participate in Demo Jam using my idea. Whilst somewhat taken aback, and to be fair pretty nervous about the consequences, they agreed to do it. I then told them my idea which was well received in concept but only added to their anxiety about how they would carry this off.
There was one emerging problem. I got the distinct impression that the staging people were not too impressed with the whole concept of having pyrotechnics inside the Sofitel Wentworth Ballroom. Personally I didn’t see the problem as I was going for a big sight and sound experience anyway so I figured the smell of singed eyebrows would only enhance the audience response but I was in somewhat of a minority. My alternate presenter came to the rescue here with a great, and practical, idea for the big finish I was after. We decided to have the drone dance to music rather than blow things up. It was a great idea and, under the circumstances, a very reasonable one as well. (I’ll have to save the pyro idea for another year.)
And then the final problem. The poor person I had roped into doing the presentation is not located anywhere near me – they are in another state. While I could send them videos, show then the code, and talk them through the idea it was impossible to rehearse unless they were in the same place as the helicopter. And the helicopter was somewhat unpredictable – it was going to be vitally important to spend a few hours getting used to its idiosyncrasies to be able to control it onstage. The last thing we wanted was for it to spear off into the audience and hit someone – that would have ruined the demo. 😉
So in the end we took the decision that I would just do the demo at the end of the Demo Jam without having it in the contest. That way I could still show off my idea but there could be no suggestion of a conflict of interest. It was the best and only reasonable outcome – and it also gave me the luxury of not having to cope with the 6 minute time limit pressure..
Special thanks here to Gregor Wolf who stepped in as my ground crew. Gregor had the fairly important task of ensuring the drone stayed within its approved airspace.