SAPUI5 - Cordova speech recognition input control

[eamarce23]

Introduction

What if we could help optimize data input as much as possible for our mobile users?

This was the thought I had when I decided trying to create a custom SAPUI5 control that could gather data from speech recognition and place it for further processing.

I have found we can make it possible by using a Cordova plugin, and by deploying our SAPUI5 application as a Hybrid application. On the case we would want to convert an already existing application and enable our control we can use the feature of Hybrid Mobile Project from Web App on the SAP Web IDE, although this feature has been marked as experimental on the SAP Web IDE Hybrid App Toolkit Add-on documentation for now.

This control will behave as a normal sap.m.input with the speech recognition button disabled on the case our application is not running on an Hybrid container.

Here is a short demo on how the final control looks:

So let's move on and see how we can create and use this control.

Prerequisites

We are going to need the following tools:

• SAP Web IDE 1.15
• Hybrid Application Toolkit 1.8.2
• A mobile device (in my case Android) for testing.
  • On this case, our Android device should be connected
  • Have the proper drivers installed
  • Have the Developer mode enabled
  • Have Android 4.4 and the Usb debugging mode enabled, if you wish to debug the project with the chrome://inspect functionality.
• Getting the macdonst/SpeechRecognitionPlugin · GitHub plugin and integrate it with HAT ( How to do this will be explained later )

Creating a SAPUI5 project to build and test the control

Ok, so first thing first, we will create a SAPUI5 project application in order to be able to write, build and test our control.

So, we should enter on the SAP Web IDE, once there we should select File>New>Project and then specify we want a SAPUI5 Application Project. We will make this project hybrid at a later stage, so we can use the Cordova plugin for voice recognition.

We shall name it CustomControlApp and proceed.

Next we can should enter our namespace ( custom.controls.demo ), select the view type as XML and we introduce our view name ( CustomControlView ).

After selecting Finish, we will see that the following project structure will be created for us:

Adding the control to our project

Once that we have our project generated, we will need to a folder named control, under the project root directory. Next we will create a new file inside this folder named SpeechRecognitionInputControl.js where we will place our custom control.

Now, we can include the following code on our SpeechRecognitionInputControl.js control file:

SpeechRecognitionInputControl.js

jQuery.sap.declare("control.SpeechRecognitionInputControl"); jQuery.sap.require("sap.m.Button"); jQuery.sap.require("sap.ui.core.Icon"); sap.m.Input.extend("control.SpeechRecognitionInputControl", { //inherit Input definition   metadata: {   properties:{   width: {type : "string", defaultValue: "70%"},   /// setting default width   value: {type : "string", defaultValue: ""},   recognition: { type:"any" }   //// for Cordova plug-in recognition object   },      aggregations : {                     _buyButton      : {type : "sap.m.Button", multiple : false, visibility: "hidden"}   // Agregate button                 }         }, init : function(){   /// init the control              var oControl = this;                 var oBuyBtn   = new sap.m.Button({                   text:"", width:"40px",  icon:"sap-icon://microphone", type:"Default",                    press: function (oEvent) {    //////// Handle press event                     if ( oControl.recognition !== undefined){                     oControl.recognition.start();  //// Start recognition                     var _oBuyBtn = oControl.getAggregation("_buyButton");         _oBuyBtn.setType(sap.m.ButtonType.Emphasized);  /// Change button color                     }                   }                 });                 this.setAggregation("_buyButton", oBuyBtn);  /// Add aggregation control                 if (sap.hybrid !== undefined ){  ///// hybrid library defined ?                 var isCompanionApp = sap.hybrid.getUrlParameterName("companionbuster");   if (window.cordova || sap.hybrid.Cordova || isCompanionApp) {   //// Verifying if it is a Cordova app                 document.addEventListener("deviceready", function() {   // load odata library   oControl.recognition = new SpeechRecognition();  /// Load Speech recognition librar   oControl.recognition.onnomatch = function(event) {         /// Add event handlers   var _oBuyBtn = oControl.getAggregation("_buyButton");         _oBuyBtn.setType(sap.m.ButtonType.Default);   };   oControl.recognition.onerror = function(event) {          /// Add event handlers   var _oBuyBtn = oControl.getAggregation("_buyButton");         _oBuyBtn.setType(sap.m.ButtonType.Default);   };     oControl.recognition.onresult = function(event) {      /// Add event handlers for result         if (event.results.length > 0) {       /// If there is a success             oControl.setValue(this.value = event.results[0][0].transcript);  /// Set value with voice recognition         }         var _oBuyBtn = oControl.getAggregation("_buyButton");         _oBuyBtn.setType(sap.m.ButtonType.Default);  /// Get back the button to original state   };   }, false);   }else{   oBuyBtn.setType(sap.m.ButtonType.Default);   oBuyBtn.setEnabled(false);   //// Disable button if there is no cordova app   }                 }else{                 /////////////// hybrid not defined                 oBuyBtn.setType(sap.m.ButtonType.Default);   oBuyBtn.setEnabled(false);   //// Disable button if there is no cordova app                 }    },    renderer : {             render : function(oRm, oControl) {   /////// Render the control             oRm.write("<div");             oRm.writeControlData(oControl);       ////// Render control data             oRm.writeStyles();             oRm.write(">");             sap.m.InputRenderer.render(oRm, oControl);  //// pass the control to base renderer             oRm.renderControl(oControl.getAggregation("_buyButton"));  /// pass aggregated control for rendering             oRm.write("</div>");         }   } });

We will not visit the whole process of creating a custom control on SAPUI5 by now, but here are a couple of blogs that may help you jump start on this process:

• How to create custom control from scratch

• Creating custom SAPUI5 controls

Basically, what we are doing on this control is inheriting from a sap.m.Input control first. By that we will have the properties and methods of this control. Next, we are aggregating a sap.m.button control to fire the Speech recognition process.We also check that our application is running on a Hybrid container, and creating a new SpeechRecognition object to use the functionality of the Cordova plugin.

Ok, so next, we will add the following code to our CustomControlView, so we can use our custom control on our sample application:

CustomControlView.view.xml

<core:View xmlns:custom="control" xmlns:core="sap.ui.core" xmlns:mvc="sap.ui.core.mvc" xmlns="sap.m"  controllerName="custom.controls.demo.view.CustomControlView">   <Page navButtonPress="onNavBack" showNavButton="false" title="Cordova Speech recognition custom control (SAPUI5)" id="thisPage">   <content>     <Button text="Add Speech recognition control" id="__button0" icon="sap-icon://add" type="Emphasized" press="onButtonPress" width="100%" />     <custom:SpeechRecognitionInputControl width="70%" id="spoken1"></custom:SpeechRecognitionInputControl>   </content>   </Page> </core:View>

From this code whe can see that we are registering our xml namespace xmlns:custom="control" and later we use it to declare our    <custom:SpeechRecognitionInputControl> control.

Next on our controller we should add the following code:

CustomControlView.controller.js
sap.ui.localResources('control'); jQuery.sap.require("control.SpeechRecognitionInputControl"); sap.ui.controller("custom.controls.demo.view.CustomControlView", {   onButtonPress: function(evt){   var x = new control.SpeechRecognitionInputControl();   var oLayout = this.getView().byId("thisPage");       oLayout.addContent(x);   } });

The important part that we can see is that we should make the location of our control known to SAPUI5. We do this by using the sap.ui.localResources('control') directive. Next we should import our custom control class with jQuery.sap.require("control.SpeechRecognitionInputControl").

Of course, we can add more instances of our control programmaticaly, and for this we include the onButtonPress event handler.

After doing this steps we could test our project to verify that our custom control is being added and our project runs. To do so, we can right click on the index.html file > Run > Run as > Web application option. At this step we will not be able to use the Speech recognition functionality yet, but we will do it by adding some extra steps next.

Adding the Speech recognition plugin to Hybrid Application Toolkit

Now we should add our Custom plugin to our HAT installation. For this, I recommend we follow the steps shown on this blog from Simmaco Ferreiro: How to use custom Cordova plugins with SAP Web IDE and Hybrid Application Toolkit . 

What we need to do is basically getting the macdonst/SpeechRecognitionPlugin · GitHub from git, on our custom plugin folder for HAT installation, by using the following command:

git clone https://github.com/macdonst/SpeechRecognitionPlugin.git

Then, when we proceed to install the HAT, we should specify the folder where we have our custom plugins and we should select our Speech recognition plugin next.

Once that we have completed the HAT installation and generation of the Companion App, we will proceed to convert our existing SAPUI5 application to an Hybrid Application project from within SAP Web IDE.

Making our SAPUI5 application an hybrid application

To complete this step, we should have enabled the Hybrid App Toolkit plugin on SAP Web IDE, and have our HAT installed and running. We can find steps to do this on the following document: How to use Hybrid Application Toolkit (HAT) and test the apps

Now, to convert our existing SAPUI5 application project to a Hybrid Mobile Project, we can right click our main folder project and follow the path New > Hybrid Mobile Project from Web App.

This is a one step wizard  where we should specify which project should be converted.

Next, we need will configure the information of the application that we are going to deploy.

To do this, we should right click our project's main folder and follow the path Project Settings > Device configuration.

We here should specify the App Name: CustomControlApp, the App ID: com.custom control, and the Version: 1.0.0.

We have to specify the platform where our application will run under the Platform section. In my case I selected Android.

Next, with our HAT running, we should right click on the Custom button, under the Plugin settings and select our SpeechRecognition plugin.

Deploying our application

Now it is time to deploy our application and testing it. For this, right click our project main folder and follow the path Deploy > Deploy to local Hybrid App Toolkit. This will deploy the application to HAT and it will create and compile our hybrid app.

Now, we can run our project by right clicking our index.html > Run on > Android device.

This will install our project's application on our device. We now can click on the microphone button and see it in action (for the cordova plugin to work as the full Android speech recognition we will need to have an active internet connection).

Debbuging our application

As a last step, we could debbug our application by using the chrome://inspect functionality. For this, we could follow the steps included on the following document: Getting Started with Kapsel - Part 3 -- AppUpdate (SP09+

Conclusion

I hope you have enjoyed this blog, I see a lot of really interesting tools and concepts that SAP is providing us, and that we can apply trying to build better experiences for our users. I hope that you find this control useful too and that you may take this idea and implement this functionality in your projects. Thanks a lot ! Happy coding :smile:

Note: You can get the full source project here: eamarce/SAPUI5CustomControlApp · GitHub

Thanks!