Unity3D and Alexa working together

If you are an SAP Employee, please follow us on Jam.

Since a long time…I had the idea of making Unity3D and Alexa work together…however…other project kept me away for actually doing it…so…a couple of days ago…a conversation with a friend made me remember that I actually really wanted to do this…so I did 🙂

At first…I wasn’t exactly sure how to do it…but then slowly the main idea came into my mind…what if Unity read a webservice that gets updated by Alexa? When the right command is parsed, then Unity will create the object and problems is solved…seems easy? Well…it actually is…

First things first…we need to create a small NodeJS webserver on Heroku…then…we need to install the Heroku Toolbelt…

Now…create a folder called node_alexa and inside create the following files…

package.json

{
  "dependencies": {
    "express": "4.13.3"
  },
  "engines": {
    "node": "0.12.7"
  }
}

procfile

web: node index.js

index.js

var express = require('express')
    ,app = express()
    ,last_value;

app.set('port', (process.env.PORT || 5000));

app.get('/', function (req, res) {
  if(req.query.command == ""){
	res.send("{ \"command\":\"" + last_value + "\"}");
  }else{
	if(req.query.command == "empty"){
		last_value = "";
		res.send("{}");
	}else{
		res.send("{ \"command\":\"" + req.query.command + "\"}");
		last_value = req.query.command;
	}
  }
})

app.listen(app.get('port'), function () {
  console.log("Node app is running on port', app.get('port')");
})

Once you have that…log into your Heroku Toolbelt and write the following…

Heroku Toolbelt

cd node_alexa
git init .
git add .
git commit -m "Init"
heroku apps:create "yourappname"
git push heroku master
heroku ps:scale = web 0
heroku ps:scale = web 1

Your webservice is ready to rock 🙂 You should be able to find by going to “http://yourappname.herokuapp.com/”

Now…this simple NodeJS powered webservice will serve as a simple Echo server…meaning…whatever you type will be returned as a json response…of course…if you type “empty” then the response will be a empty json…so the main idea here is that we can keep the last entered value…if you pass a command it will be called again when you don’t pass any commands at all…so by calling it once…we can cal it multiple times without disrupting its value…

Next in line…will be to create our Unity app…

Create a new app and call it “WebService” or something like that…project name doesn’t matter too much…

If the Hierarchy window select “Main Camera” and change the “Tranform” details like this…

Now, create a new “3D Object” -> “Cube” and name it “Platform” with the following “Transform” details…

After that, we might need to create four wall that will go around the platform…so create 4 “3D Object” -> “Cube” and name them “Wall 1”, “Wall 2”, “Wall 3” and “Wall 4″…

When everything is ready, your workspace should look like this…

Go to the project tab and create a new folder called “plugins” and then create a new C# file called “SimpleJSON”…inside copy the source code from here…this will allow us to use SimpleJSON to parse the JSON…

Now…create another folder called “Script” and inside create a new C# file called “MetaCoding”…or whatever you like…

MetaCoding.cs

using UnityEngine;
using System.Collections;
using System.Net;
using System.IO;
using SimpleJSON;

public class MetaCoding : MonoBehaviour {

    int counter = 1;

    IEnumerator DownloadWebService()
    {
        while (true) { 
            WWW w = new WWW("http://yourapp.herokuapp.com/?command");
            yield return w;

            print("Waiting for webservice\n");

            yield return new WaitForSeconds(1f);

            print("Received webservice\n");
        
            ExtractCommand(w.text);

            print("Extracted information");

            WWW y = new WWW("http://yourapp.herokuapp.com/?command=empty");
            yield return y;

            print("Cleaned webservice");

            yield return new WaitForSeconds(5);
        }
    }

    void ExtractCommand(string json)
    {
        var jsonstring = JSON.Parse(json);
        string command = jsonstring["command"];
        print(command);
        if (command == null) { return;  }
        string[] commands_array = command.Split(" "[0]);
        if(commands_array.Length < 3)
        {
            return;
        }
        if (commands_array[0] == "create")
        {
            CreateObject(commands_array[1], commands_array[2]);
        }
    }

    void CreateObject(string color, string shape)
    {

        string name = "NewObject_" + counter;
        counter += 1;
        GameObject NewObject = new GameObject(name);

        switch (shape)
        {
            case "cube":
                NewObject = GameObject.CreatePrimitive(PrimitiveType.Cube);
                break;
            case "sphere":
                NewObject = GameObject.CreatePrimitive(PrimitiveType.Sphere);
                break;
            case "cylinder":
                NewObject = GameObject.CreatePrimitive(PrimitiveType.Cylinder);
                break;
            case "capsule":
                NewObject = GameObject.CreatePrimitive(PrimitiveType.Capsule);
                break;
        }
        NewObject.transform.position = new Vector3(0, 5, 0);
        NewObject.AddComponent<Rigidbody>();
        switch (color)
        {
            case "red":
                NewObject.GetComponent<Renderer>().material.color = Color.red;
                break;
            case "yellow":
                NewObject.GetComponent<Renderer>().material.color = Color.yellow;
                break;
            case "green":
                NewObject.GetComponent<Renderer>().material.color = Color.green;
                break;
            case "blue":
                NewObject.GetComponent<Renderer>().material.color = Color.blue;
                break;
            case "black":
                NewObject.GetComponent<Renderer>().material.color = Color.black;
                break;
            case "white":
                NewObject.GetComponent<Renderer>().material.color = Color.white;
                break;
        }
    }

        // Use this for initialization
    void Start () {
        print("Started webservice import...\n");

        StartCoroutine(DownloadWebService());
    }
	
	// Update is called once per frame
	void Update () {
	
	}
}

Once you have the code…simply attach the script to the Main Camera…

The basic concept for this script is pretty simple…We’re creating “DownloadWebService” as an IEnumerator method so we can call it as a Coroutine…and that allow us to have a sleep as we want to give some time between calls…

This method will fetch our Heroku WebService looking for a “create” command…once it has it…it will parse the JSON response and split in 3…so we can have…”create”, “blue” and “sphere”…this will call CreateObject which will then create a blue sphere…after we have done that…the coroutine will continue as simply send a new command to our WebService to clean the output…to make this work nicely…we want to give 5 seconds after we clean the webservice before trying to see if there’s another “create” call…

And this call be will be done by our Alexa skill…so basically when saying “create blue sphere” on Alexa…she will be send the command to the WebService…update the message and our Unity app will grab it…do its work…and clean up the Webservice…the wait for Alexa to provide the next command…

So…to kind of wrap up…we need to create our Alexa skill…

First, we’re going to create a Lambda function…so log in here…

Of course…I have everything already setup…so I’m going to create a dummy function just to show the steps…

Click on “Create Lambda Function” and you will be presented with this…

There’s a bunch of course…so type in “Color” in the filter box…

Choose “alexa-skills-kit-color-expert”

Leave this as it is and press “Next”

Choose a name and a description…

Choose an existing role if you have it already…otherwise just create a lambda_basic_execution…then raise up Timeout to 10 seconds and leave everything else as it is…press “Next”…a confirmation window will appear…so just press “Create function”…

You will be presented with a screen where you can upload your source code (which will be doing later on) and an ARN number…which we need for the next step…

The following part deals with create the Alexa skill…so please follow along…and log in here…

Choose “Alexa Skills Kit”…and create a new skill…

Choose a name for you skill and the most important…choose an “Invocation Name”…which is what you’re going to use tell Alexa to open you application…something like…”Alexa, open Sandbox”…click next…

On the Interaction Model tab we have two windows…fill this on “Intent Schema”…

Intent Schema

{
  "intents": [
    {
      "intent": "GetUnityIntent",
      "slots": [
        {
          "name": "color",
          "type": "LITERAL"        	
        },
        {
          "name": "shape",
          "type": "LITERAL"
        }
      ]
    },
    {
      "intent": "HelpIntent",
      "slots": []
    }
  ]
}

This are basically the parameters that we can use when asking Alexa to do something…

And fill this on “Sample Utterances”…

Sample Utterances

GetUnityIntent create {red|color} {sphere|shape}
GetUnityIntent create {yellow|color} {sphere|shape}
GetUnityIntent create {green|color} {sphere|shape}
GetUnityIntent create {blue|color} {sphere|shape}
GetUnityIntent create {black|color} {sphere|shape}
GetUnityIntent create {white|color} {sphere|shape}

GetUnityIntent create {red|color} {cube|shape}
GetUnityIntent create {yellow|color} {cube|shape}
GetUnityIntent create {green|color} {cube|shape}
GetUnityIntent create {blue|color} {cube|shape}
GetUnityIntent create {black|color} {cube|shape}
GetUnityIntent create {white|color} {cube|shape}

GetUnityIntent create {red|color} {cylinder|shape}
GetUnityIntent create {yellow|color} {cylinder|shape}
GetUnityIntent create {green|color} {cylinder|shape}
GetUnityIntent create {blue|color} {cylinder|shape}
GetUnityIntent create {black|color} {cylinder|shape}
GetUnityIntent create {white|color} {cylinder|shape}

GetUnityIntent create {red|color} {capsule|shape}
GetUnityIntent create {yellow|color} {capsule|shape}
GetUnityIntent create {green|color} {capsule|shape}
GetUnityIntent create {blue|color} {capsule|shape}
GetUnityIntent create {black|color} {capsule|shape}
GetUnityIntent create {white|color} {capsule|shape}

GetUnityIntent {thank you|color}

This are all the commands that Alexa can understand…and yes…we could have used “Custom Slot Types” to make the code shorter…but…I have had the problems of not working pretty well with more than one slot…simply hit next…

Here, choose AWS Lambda ARN…and pick either North America or Europe depending on your physical location…the on the text box…simply copy and paste the ARN that you received from your Lambda function…

This will send you to the “Test” tab…but we don’t want to and actually we can’t use that yet…so go back to the “Skill Information” tab and you will find that a new field has appeared…

And that should be “Application Id”…copy this number and let’s move on to the final step…

Create a folder called “Unity” and inside a folder called “src”…inside that folder copy this file “AlexaSkills.js”

We’re going to use the “request” module of NodeJS…so install it locally on the Unity folder like this…

sudo npm install --prefix=~/Unity/src request

This will create a node_module folder with the request module on it…

Then, create a new file called “index.js”

index.js

var request = require("request")
  , AlexaSkill = require('./AlexaSkill')
    , APP_ID     = 'yourappid';

var error = function (err, response, body) {
    console.log('ERROR [%s]', err);
};

var getJsonFromUnity = function(color, shape, callback){

var command = "create " + color + " " + shape;

if(color == "thank you"){
	callback("thank you");
}
else{
var options = { method: 'GET',
  url: 'http://yourapp.herokuapp.com/',
  qs: { command: command },
  headers: 
   { 'postman-token': '230914f7-c478-4f13-32fd-e6593d8db4d1',
     'cache-control': 'no-cache' } };

var error_log = "";

request(options, function (error, response, body) {
	if (!error) {
		error_log = color + " " + shape;
	}else{
		error_log = "There was a mistake";
	}
		callback(error_log);
    });
}
}

var handleUnityRequest = function(intent, session, response){
  getJsonFromUnity(intent.slots.color.value,intent.slots.shape.value, function(data){
	if(data != "thank you"){
	var text = 'The ' + data + ' has been created';
	var reprompt = 'Which shape would you like?';
    response.ask(text, reprompt);
	}else{
		response.tell("You're welcome");
	}
  });
};

var Unity = function(){
  AlexaSkill.call(this, APP_ID);
};

Unity.prototype = Object.create(AlexaSkill.prototype);
Unity.prototype.constructor = Unity;

Unity.prototype.eventHandlers.onSessionStarted = function(sessionStartedRequest, session){
  console.log("onSessionStarted requestId: " + sessionStartedRequest.requestId
      + ", sessionId: " + session.sessionId);
};

Unity.prototype.eventHandlers.onLaunch = function(launchRequest, session, response){
  // This is when they launch the skill but don't specify what they want.

  var output = 'Welcome to Unity. Create any color shape by saying create and providing a color and a shape';

  var reprompt = 'Which shape would you like?';

  response.ask(output, reprompt);

  console.log("onLaunch requestId: " + launchRequest.requestId
      + ", sessionId: " + session.sessionId);
};

Unity.prototype.intentHandlers = {
  GetUnityIntent: function(intent, session, response){
    handleUnityRequest(intent, session, response);
  },

  HelpIntent: function(intent, session, response){
    var speechOutput = 'Create a new colored shape. Which shape would you like?';
    response.ask(speechOutput);
  }
};

exports.handler = function(event, context) {
    var skill = new Unity();
    skill.execute(event, context);
};

This code is very simple…because it mostly a template…you simply copy it…change a couple of things and you’re ready to go…

Basically when you say “Alexa, open Unity”…she will listen for your requests…so you can say “create green cube”…so will call our Heroku WebService and the wait for another command…if you doesn’t speak to her again…she will prompt you to say something…if you say “Thank you” she will politely deactivate herself…

And that’s pretty much it…once Alexa send the command to the WebServer…our Unity App will read and act accordingly…creating whatever shape and color you requested…nice, huh?

But of course…you don’t believe, don’t you? It can’t be that simple…well…yes and no…it’s simple…but I took all the pain point and provide you with the nice and clean set of instructions…

So…here’s how it looks like when you run the Unity app…

And here the action video…

Hope you like it…and stay tuned…because for me this was only a proof of concept…the real thing will become my next full time project…

– Your d-shop team