Unity3D and Alexa working together
If you are an SAP Employee, please follow us on Jam.
Since a long time…I had the idea of making Unity3D and Alexa work together…however…other project kept me away for actually doing it…so…a couple of days ago…a conversation with a friend made me remember that I actually really wanted to do this…so I did 🙂
At first…I wasn’t exactly sure how to do it…but then slowly the main idea came into my mind…what if Unity read a webservice that gets updated by Alexa? When the right command is parsed, then Unity will create the object and problems is solved…seems easy? Well…it actually is…
First things first…we need to create a small NodeJS webserver on Heroku…then…we need to install the Heroku Toolbelt…
Now…create a folder called node_alexa and inside create the following files…
package.json
{
"dependencies": {
"express": "4.13.3"
},
"engines": {
"node": "0.12.7"
}
}
procfile
web: node index.js
index.js
var express = require('express')
,app = express()
,last_value;
app.set('port', (process.env.PORT || 5000));
app.get('/', function (req, res) {
if(req.query.command == ""){
res.send("{ \"command\":\"" + last_value + "\"}");
}else{
if(req.query.command == "empty"){
last_value = "";
res.send("{}");
}else{
res.send("{ \"command\":\"" + req.query.command + "\"}");
last_value = req.query.command;
}
}
})
app.listen(app.get('port'), function () {
console.log("Node app is running on port', app.get('port')");
})
Once you have that…log into your Heroku Toolbelt and write the following…
Heroku Toolbelt
cd node_alexa
git init .
git add .
git commit -m "Init"
heroku apps:create "yourappname"
git push heroku master
heroku ps:scale = web 0
heroku ps:scale = web 1
Your webservice is ready to rock 🙂 You should be able to find by going to “http://yourappname.herokuapp.com/”
Now…this simple NodeJS powered webservice will serve as a simple Echo server…meaning…whatever you type will be returned as a json response…of course…if you type “empty” then the response will be a empty json…so the main idea here is that we can keep the last entered value…if you pass a command it will be called again when you don’t pass any commands at all…so by calling it once…we can cal it multiple times without disrupting its value…
Next in line…will be to create our Unity app…
Create a new app and call it “WebService” or something like that…project name doesn’t matter too much…
If the Hierarchy window select “Main Camera” and change the “Tranform” details like this…
Now, create a new “3D Object” -> “Cube” and name it “Platform” with the following “Transform” details…
After that, we might need to create four wall that will go around the platform…so create 4 “3D Object” -> “Cube” and name them “Wall 1”, “Wall 2”, “Wall 3” and “Wall 4″…
When everything is ready, your workspace should look like this…
Go to the project tab and create a new folder called “plugins” and then create a new C# file called “SimpleJSON”…inside copy the source code from here…this will allow us to use SimpleJSON to parse the JSON…
Now…create another folder called “Script” and inside create a new C# file called “MetaCoding”…or whatever you like…
MetaCoding.cs
using UnityEngine;
using System.Collections;
using System.Net;
using System.IO;
using SimpleJSON;
public class MetaCoding : MonoBehaviour {
int counter = 1;
IEnumerator DownloadWebService()
{
while (true) {
WWW w = new WWW("http://yourapp.herokuapp.com/?command");
yield return w;
print("Waiting for webservice\n");
yield return new WaitForSeconds(1f);
print("Received webservice\n");
ExtractCommand(w.text);
print("Extracted information");
WWW y = new WWW("http://yourapp.herokuapp.com/?command=empty");
yield return y;
print("Cleaned webservice");
yield return new WaitForSeconds(5);
}
}
void ExtractCommand(string json)
{
var jsonstring = JSON.Parse(json);
string command = jsonstring["command"];
print(command);
if (command == null) { return; }
string[] commands_array = command.Split(" "[0]);
if(commands_array.Length < 3)
{
return;
}
if (commands_array[0] == "create")
{
CreateObject(commands_array[1], commands_array[2]);
}
}
void CreateObject(string color, string shape)
{
string name = "NewObject_" + counter;
counter += 1;
GameObject NewObject = new GameObject(name);
switch (shape)
{
case "cube":
NewObject = GameObject.CreatePrimitive(PrimitiveType.Cube);
break;
case "sphere":
NewObject = GameObject.CreatePrimitive(PrimitiveType.Sphere);
break;
case "cylinder":
NewObject = GameObject.CreatePrimitive(PrimitiveType.Cylinder);
break;
case "capsule":
NewObject = GameObject.CreatePrimitive(PrimitiveType.Capsule);
break;
}
NewObject.transform.position = new Vector3(0, 5, 0);
NewObject.AddComponent<Rigidbody>();
switch (color)
{
case "red":
NewObject.GetComponent<Renderer>().material.color = Color.red;
break;
case "yellow":
NewObject.GetComponent<Renderer>().material.color = Color.yellow;
break;
case "green":
NewObject.GetComponent<Renderer>().material.color = Color.green;
break;
case "blue":
NewObject.GetComponent<Renderer>().material.color = Color.blue;
break;
case "black":
NewObject.GetComponent<Renderer>().material.color = Color.black;
break;
case "white":
NewObject.GetComponent<Renderer>().material.color = Color.white;
break;
}
}
// Use this for initialization
void Start () {
print("Started webservice import...\n");
StartCoroutine(DownloadWebService());
}
// Update is called once per frame
void Update () {
}
}
Once you have the code…simply attach the script to the Main Camera…
The basic concept for this script is pretty simple…We’re creating “DownloadWebService” as an IEnumerator method so we can call it as a Coroutine…and that allow us to have a sleep as we want to give some time between calls…
This method will fetch our Heroku WebService looking for a “create” command…once it has it…it will parse the JSON response and split in 3…so we can have…”create”, “blue” and “sphere”…this will call CreateObject which will then create a blue sphere…after we have done that…the coroutine will continue as simply send a new command to our WebService to clean the output…to make this work nicely…we want to give 5 seconds after we clean the webservice before trying to see if there’s another “create” call…
And this call be will be done by our Alexa skill…so basically when saying “create blue sphere” on Alexa…she will be send the command to the WebService…update the message and our Unity app will grab it…do its work…and clean up the Webservice…the wait for Alexa to provide the next command…
So…to kind of wrap up…we need to create our Alexa skill…
First, we’re going to create a Lambda function…so log in here…
Of course…I have everything already setup…so I’m going to create a dummy function just to show the steps…
Click on “Create Lambda Function” and you will be presented with this…
There’s a bunch of course…so type in “Color” in the filter box…
Choose “alexa-skills-kit-color-expert”
Leave this as it is and press “Next”
Choose a name and a description…
Choose an existing role if you have it already…otherwise just create a lambda_basic_execution…then raise up Timeout to 10 seconds and leave everything else as it is…press “Next”…a confirmation window will appear…so just press “Create function”…
You will be presented with a screen where you can upload your source code (which will be doing later on) and an ARN number…which we need for the next step…
The following part deals with create the Alexa skill…so please follow along…and log in here…
Choose “Alexa Skills Kit”…and create a new skill…
Choose a name for you skill and the most important…choose an “Invocation Name”…which is what you’re going to use tell Alexa to open you application…something like…”Alexa, open Sandbox”…click next…
On the Interaction Model tab we have two windows…fill this on “Intent Schema”…
Intent Schema
{
"intents": [
{
"intent": "GetUnityIntent",
"slots": [
{
"name": "color",
"type": "LITERAL"
},
{
"name": "shape",
"type": "LITERAL"
}
]
},
{
"intent": "HelpIntent",
"slots": []
}
]
}
This are basically the parameters that we can use when asking Alexa to do something…
And fill this on “Sample Utterances”…
Sample Utterances
GetUnityIntent create {red|color} {sphere|shape}
GetUnityIntent create {yellow|color} {sphere|shape}
GetUnityIntent create {green|color} {sphere|shape}
GetUnityIntent create {blue|color} {sphere|shape}
GetUnityIntent create {black|color} {sphere|shape}
GetUnityIntent create {white|color} {sphere|shape}
GetUnityIntent create {red|color} {cube|shape}
GetUnityIntent create {yellow|color} {cube|shape}
GetUnityIntent create {green|color} {cube|shape}
GetUnityIntent create {blue|color} {cube|shape}
GetUnityIntent create {black|color} {cube|shape}
GetUnityIntent create {white|color} {cube|shape}
GetUnityIntent create {red|color} {cylinder|shape}
GetUnityIntent create {yellow|color} {cylinder|shape}
GetUnityIntent create {green|color} {cylinder|shape}
GetUnityIntent create {blue|color} {cylinder|shape}
GetUnityIntent create {black|color} {cylinder|shape}
GetUnityIntent create {white|color} {cylinder|shape}
GetUnityIntent create {red|color} {capsule|shape}
GetUnityIntent create {yellow|color} {capsule|shape}
GetUnityIntent create {green|color} {capsule|shape}
GetUnityIntent create {blue|color} {capsule|shape}
GetUnityIntent create {black|color} {capsule|shape}
GetUnityIntent create {white|color} {capsule|shape}
GetUnityIntent {thank you|color}
This are all the commands that Alexa can understand…and yes…we could have used “Custom Slot Types” to make the code shorter…but…I have had the problems of not working pretty well with more than one slot…simply hit next…
Here, choose AWS Lambda ARN…and pick either North America or Europe depending on your physical location…the on the text box…simply copy and paste the ARN that you received from your Lambda function…
This will send you to the “Test” tab…but we don’t want to and actually we can’t use that yet…so go back to the “Skill Information” tab and you will find that a new field has appeared…
And that should be “Application Id”…copy this number and let’s move on to the final step…
Create a folder called “Unity” and inside a folder called “src”…inside that folder copy this file “AlexaSkills.js”
We’re going to use the “request” module of NodeJS…so install it locally on the Unity folder like this…
sudo npm install --prefix=~/Unity/src request
This will create a node_module folder with the request module on it…
Then, create a new file called “index.js”
index.js
var request = require("request")
, AlexaSkill = require('./AlexaSkill')
, APP_ID = 'yourappid';
var error = function (err, response, body) {
console.log('ERROR [%s]', err);
};
var getJsonFromUnity = function(color, shape, callback){
var command = "create " + color + " " + shape;
if(color == "thank you"){
callback("thank you");
}
else{
var options = { method: 'GET',
url: 'http://yourapp.herokuapp.com/',
qs: { command: command },
headers:
{ 'postman-token': '230914f7-c478-4f13-32fd-e6593d8db4d1',
'cache-control': 'no-cache' } };
var error_log = "";
request(options, function (error, response, body) {
if (!error) {
error_log = color + " " + shape;
}else{
error_log = "There was a mistake";
}
callback(error_log);
});
}
}
var handleUnityRequest = function(intent, session, response){
getJsonFromUnity(intent.slots.color.value,intent.slots.shape.value, function(data){
if(data != "thank you"){
var text = 'The ' + data + ' has been created';
var reprompt = 'Which shape would you like?';
response.ask(text, reprompt);
}else{
response.tell("You're welcome");
}
});
};
var Unity = function(){
AlexaSkill.call(this, APP_ID);
};
Unity.prototype = Object.create(AlexaSkill.prototype);
Unity.prototype.constructor = Unity;
Unity.prototype.eventHandlers.onSessionStarted = function(sessionStartedRequest, session){
console.log("onSessionStarted requestId: " + sessionStartedRequest.requestId
+ ", sessionId: " + session.sessionId);
};
Unity.prototype.eventHandlers.onLaunch = function(launchRequest, session, response){
// This is when they launch the skill but don't specify what they want.
var output = 'Welcome to Unity. Create any color shape by saying create and providing a color and a shape';
var reprompt = 'Which shape would you like?';
response.ask(output, reprompt);
console.log("onLaunch requestId: " + launchRequest.requestId
+ ", sessionId: " + session.sessionId);
};
Unity.prototype.intentHandlers = {
GetUnityIntent: function(intent, session, response){
handleUnityRequest(intent, session, response);
},
HelpIntent: function(intent, session, response){
var speechOutput = 'Create a new colored shape. Which shape would you like?';
response.ask(speechOutput);
}
};
exports.handler = function(event, context) {
var skill = new Unity();
skill.execute(event, context);
};
This code is very simple…because it mostly a template…you simply copy it…change a couple of things and you’re ready to go…
Basically when you say “Alexa, open Unity”…she will listen for your requests…so you can say “create green cube”…so will call our Heroku WebService and the wait for another command…if you doesn’t speak to her again…she will prompt you to say something…if you say “Thank you” she will politely deactivate herself…
And that’s pretty much it…once Alexa send the command to the WebServer…our Unity App will read and act accordingly…creating whatever shape and color you requested…nice, huh?
But of course…you don’t believe, don’t you? It can’t be that simple…well…yes and no…it’s simple…but I took all the pain point and provide you with the nice and clean set of instructions…
So…here’s how it looks like when you run the Unity app…
And here the action video…
Hope you like it…and stay tuned…because for me this was only a proof of concept…the real thing will become my next full time project…
– Your d-shop team
Hi Alvaro, thank you for the tutorial. I've had some issues with the AlexaSkills.js code, as the code on github you link to doesn't seem to match the rest of the tutorial. Is the link wrong, or am I missing something? Thanks!
Hi Alvaro, did you get the chance to look over my earlier comment? I'm still hoping to get this up and running on my end, please let me know what the deal is with the link. Thanks!
Hello Alex:
Sorry...I didn't see your comment 🙁
The AlexaSkills.js code is simply a set of functions that helps you to create the Alexa Skill...basically you create a folder and copy the AlexaSkills.js file along with your index.js file...then you zip both files and upload them to your Amazon Lambda function...
This blog might help you up https://blogs.sap.com/2015/09/10/amazon-alexa-and-sap-hana/
Greetings,
Blag.
Development Culture.
Hi Alvaro,
Everything worked well before checking the url :http://yourappname.herokuapp.com/ as you suggested.
When i enter my target url that is - http://mytestapp1307.herokuapp.com/ , i get a blank webpage with this message-- {"command":"undefined"}
Any help will be useful.
Thanks.
Anupam
Hello Anupam:
That's perfectly normal 🙂 And it means that the server doesn't have any default value...you could either send one command throughout Alexa or simply call it from the browser like this...
http://mytestapp1307.herokuapp.com/?command=empty
Greetings,
Blag.
Development Culture.
Thanks for your prompt response. 🙂
Hi Alvaro,
I can’t get this project running, whenever i test it on the aws lambda i get this error:
And log it says this:
I've tried every possible soultion even zipping the contents rather than the subfolder directory but it still doesn't work. Could you tell me what i'm doing wrong? thanks!
Well...if you have the .zip file with index.js and AlexaSkill.js and nothing else...then it should work...otherwise I don't know...
Greetings,
Blag.
Development Culture.
hello im trying some stuff out with alexa and unity but when im following this tutorial keep getting stuck on why there is no code for the lambda function. so there is no connection to the nodeJS server. Where do you make the connection to the server and where do you use the sudo command. Im completly stuck please help
Hello Martin:
The code for the Lambda function is the index.js file...and no connection to the nodeJS server is needed because Lambda is basically a nodeJS server...
I think the blog is pretty detailed and well explained...but...maybe you can start with this...
https://developer.amazon.com/alexa-skills-kit/tutorials/fact-skill-1?&sc_channel=SEM&sc_campaign=Fact-Skill&sc_detail=Branded&sc_segment=Alexa-Tutorial&sc_publisher=Google&sc_country=WW&sc_medium=SEM_Fact-Skill_Branded_Alexa-Tutorial_Google_WW_0007&sc_trackingcode=0007&gclid=EAIaIQobChMIv5ax-LC21gIVFLbACh2n6wagEAAYASAAEgJFofD_BwE
Greetings,
Blag.
Development Culture.
We successfully make it but we have to wait it too long for unity to respond. And the voice has been come out first from Alexa. Eventually, the object created in unity and the voice are no way to be turn up at the same time.
So any solution to make them come up at the same time?
Hello Edwin:
Well...that's kind of hard to tell...if you're using Heroku like I did, then the server can fail unless you're paying for it...free versions are not the most reliable...
Also, Unity connection might be having some problems maybe because of a Firewall? Slow connection? No idea...
As you can see in my video...the response from both Alexa and Unity are pretty fast...and I have way more complex scenarios than this one where everything works perfectly...unless...there's a glitch in the Internet connection or Heroku fails for some reason...
Greetings,
Blag.
Development Culture.
HI man! For a school project in my programming/robotics class I am doing your project here to give me a base for a game i wanna make, I am having trouble with the alexa edit tool, when i paste your code into the intent shema, and the sample utterances, I get error codes. For the intent schema it says "intents dosent specify a name"and for sample utterances I says im not using one of their characters... Hopefully you can get back to me quickly!Cheers