Skip to Content
Author's profile photo EDGAR MARTINEZ

SAPUI5 – Cordova speech recognition input control


Introduction

What if we could help optimize data input as much as possible for our mobile users?

This was the thought I had when I decided trying to create a custom SAPUI5 control that could gather data from speech recognition and place it for further processing.


I have found we can make it possible by using a Cordova plugin, and by deploying our SAPUI5 application as a Hybrid application. On the case we would want to convert an already existing application and enable our control we can use the feature of Hybrid Mobile Project from Web App on the SAP Web IDE, although this feature has been marked as experimental on the SAP Web IDE Hybrid App Toolkit Add-on documentation for now.

This control will behave as a normal sap.m.input with the speech recognition button disabled on the case our application is not running on an Hybrid container.

Here is a short demo on how the final control looks:

So let’s move on and see how we can create and use this control.

Prerequisites

We are going to need the following tools:

  • SAP Web IDE 1.15
  • Hybrid Application Toolkit 1.8.2
  • A mobile device (in my case Android) for testing.
    • On this case, our Android device should be connected
    • Have the proper drivers installed
    • Have the Developer mode enabled
    • Have Android 4.4 and the Usb debugging mode enabled, if you wish to debug the project with the chrome://inspect functionality.
  • Getting the macdonst/SpeechRecognitionPlugin · GitHub plugin and integrate it with HAT ( How to do this will be explained later )


Creating a SAPUI5 project to build and test the control

Ok, so first thing first, we will create a SAPUI5 project application in order to be able to write, build and test our control.

So, we should enter on the SAP Web IDE, once there we should select File>New>Project and then specify we want a SAPUI5 Application Project. We will make this project hybrid at a later stage, so we can use the Cordova plugin for voice recognition.

1.SAPUI5 application project.png

We shall name it CustomControlApp and proceed.

2.CustomControlAPP.png


Next we can should enter our namespace ( custom.controls.demo ), select the view type as XML and we introduce our view name ( CustomControlView ).

3.template_Customization.png

After selecting Finish, we will see that the following project structure will be created for us:

/wp-content/uploads/2015/10/project_structure1_800812.png


Adding the control to our project

Once that we have our project generated, we will need to a folder named control, under the project root directory. Next we will create a new file inside this folder named SpeechRecognitionInputControl.js where we will place our custom control.

/wp-content/uploads/2015/10/project_structure2_800813.png

Now, we can include the following code on our SpeechRecognitionInputControl.js control file:

SpeechRecognitionInputControl.js

jQuery.sap.declare("control.SpeechRecognitionInputControl");
jQuery.sap.require("sap.m.Button");
jQuery.sap.require("sap.ui.core.Icon");
sap.m.Input.extend("control.SpeechRecognitionInputControl", { //inherit Input definition
  metadata: {
  properties:{
  width: {type : "string", defaultValue: "70%"},   /// setting default width
  value: {type : "string", defaultValue: ""},
  recognition: { type:"any" }   //// for Cordova plug-in recognition object
  },
     aggregations : {
                    _buyButton      : {type : "sap.m.Button", multiple : false, visibility: "hidden"}   // Agregate button
                }
        },
init : function(){   /// init the control
             var oControl = this;
                var oBuyBtn   = new sap.m.Button({
                  text:"", width:"40px",  icon:"sap-icon://microphone", type:"Default",
                   press: function (oEvent) {    //////// Handle press event
                    if ( oControl.recognition !== undefined){
                    oControl.recognition.start();  //// Start recognition
                    var _oBuyBtn = oControl.getAggregation("_buyButton");
        _oBuyBtn.setType(sap.m.ButtonType.Emphasized);  /// Change button color
                    }
                  }
                });
                this.setAggregation("_buyButton", oBuyBtn);  /// Add aggregation control
                if (sap.hybrid !== undefined ){  ///// hybrid library defined ?
                var isCompanionApp = sap.hybrid.getUrlParameterName("companionbuster");
  if (window.cordova || sap.hybrid.Cordova || isCompanionApp) {   //// Verifying if it is a Cordova app
                document.addEventListener("deviceready", function() {
  // load odata library
  oControl.recognition = new SpeechRecognition();  /// Load Speech recognition librar
  oControl.recognition.onnomatch = function(event) {         /// Add event handlers
  var _oBuyBtn = oControl.getAggregation("_buyButton");
        _oBuyBtn.setType(sap.m.ButtonType.Default);
  };
  oControl.recognition.onerror = function(event) {          /// Add event handlers
  var _oBuyBtn = oControl.getAggregation("_buyButton");
        _oBuyBtn.setType(sap.m.ButtonType.Default);
  };
    oControl.recognition.onresult = function(event) {      /// Add event handlers for result
        if (event.results.length > 0) {       /// If there is a success
            oControl.setValue(this.value = event.results[0][0].transcript);  /// Set value with voice recognition
        }
        var _oBuyBtn = oControl.getAggregation("_buyButton");
        _oBuyBtn.setType(sap.m.ButtonType.Default);  /// Get back the button to original state
  };
  }, false);
  }else{
  oBuyBtn.setType(sap.m.ButtonType.Default);
  oBuyBtn.setEnabled(false);   //// Disable button if there is no cordova app
  }
                }else{
                /////////////// hybrid not defined
                oBuyBtn.setType(sap.m.ButtonType.Default);
  oBuyBtn.setEnabled(false);   //// Disable button if there is no cordova app
                }
   },
   renderer : {
            render : function(oRm, oControl) {   /////// Render the control
            oRm.write("<div");
            oRm.writeControlData(oControl);       ////// Render control data
            oRm.writeStyles();
            oRm.write(">");
            sap.m.InputRenderer.render(oRm, oControl);  //// pass the control to base renderer
            oRm.renderControl(oControl.getAggregation("_buyButton"));  /// pass aggregated control for rendering
            oRm.write("</div>");
        }
  }
});
















We will not visit the whole process of creating a custom control on SAPUI5 by now, but here are a couple of blogs that may help you jump start on this process:

Basically, what we are doing on this control is inheriting from a sap.m.Input control first. By that we will have the properties and methods of this control. Next, we are aggregating a sap.m.button control to fire the Speech recognition process.We also check that our application is running on a Hybrid container, and creating a new SpeechRecognition object to use the functionality of the Cordova plugin.


Ok, so next, we will add the following code to our CustomControlView, so we can use our custom control on our sample application:

CustomControlView.view.xml

<core:View xmlns:custom="control" xmlns:core="sap.ui.core"
 xmlns:mvc="sap.ui.core.mvc" xmlns="sap.m"  controllerName="custom.controls.demo.view.CustomControlView">
  <Page navButtonPress="onNavBack" showNavButton="false" title="Cordova Speech recognition custom control (SAPUI5)" id="thisPage">
  <content>
    <Button text="Add Speech recognition control" id="__button0" icon="sap-icon://add" type="Emphasized" press="onButtonPress" width="100%" />
    <custom:SpeechRecognitionInputControl width="70%" id="spoken1"></custom:SpeechRecognitionInputControl>
  </content>
  </Page>
</core:View>




















From this code whe can see that we are registering our xml namespace xmlns:custom=”control” and later we use it to declare our    <custom:SpeechRecognitionInputControl> control.

Next on our controller we should add the following code:

CustomControlView.controller.js

sap.ui.localResources('control');
jQuery.sap.require("control.SpeechRecognitionInputControl");
sap.ui.controller("custom.controls.demo.view.CustomControlView", {
  onButtonPress: function(evt){
  var x = new control.SpeechRecognitionInputControl();
  var oLayout = this.getView().byId("thisPage");
      oLayout.addContent(x);
  }
});
















The important part that we can see is that we should make the location of our control known to SAPUI5. We do this by using the sap.ui.localResources(‘control’) directive. Next we should import our custom control class with jQuery.sap.require(“control.SpeechRecognitionInputControl”).

Of course, we can add more instances of our control programmaticaly, and for this we include the onButtonPress event handler.

After doing this steps we could test our project to verify that our custom control is being added and our project runs. To do so, we can right click on the index.html file > Run > Run as > Web application option. At this step we will not be able to use the Speech recognition functionality yet, but we will do it by adding some extra steps next.

/wp-content/uploads/2015/10/8_contro_running_800038.png

Adding the Speech recognition plugin to Hybrid Application Toolkit



Now we should add our Custom plugin to our HAT installation. For this, I recommend we follow the steps shown on this blog from Simmaco Ferreiro: How to use custom Cordova plugins with SAP Web IDE and Hybrid Application Toolkit . 


What we need to do is basically getting the macdonst/SpeechRecognitionPlugin · GitHub from git, on our custom plugin folder for HAT installation, by using the following command:



git clone https://github.com/macdonst/SpeechRecognitionPlugin.git



/wp-content/uploads/2015/10/plugin_install_800093.png


Then, when we proceed to install the HAT, we should specify the folder where we have our custom plugins and we should select our Speech recognition plugin next.


install HAT  plug in.png

Once that we have completed the HAT installation and generation of the Companion App, we will proceed to convert our existing SAPUI5 application to an Hybrid Application project from within SAP Web IDE.

Making our SAPUI5 application an hybrid application

To complete this step, we should have enabled the Hybrid App Toolkit plugin on SAP Web IDE, and have our HAT installed and running. We can find steps to do this on the following document: How to use Hybrid Application Toolkit (HAT) and test the apps

Now, to convert our existing SAPUI5 application project to a Hybrid Mobile Project, we can right click our main folder project and follow the path New > Hybrid Mobile Project from Web App.

9.HybridMobileProjectFromWebApp.png

This is a one step wizard  where we should specify which project should be converted.

10.CustomControlApp_Hybrid.png


Next, we need will configure the information of the application that we are going to deploy.

To do this, we should right click our project’s main folder and follow the path Project Settings > Device configuration.

We here should specify the App Name: CustomControlApp, the App ID: com.custom control, and the Version: 1.0.0.

We have to specify the platform where our application will run under the Platform section. In my case I selected Android.

12.Device_Configuration.png

Next, with our HAT running, we should right click on the Custom button, under the Plugin settings and select our SpeechRecognition plugin.

/wp-content/uploads/2015/10/11_custom_plugin_from_webide_800102.png

Deploying our application

Now it is time to deploy our application and testing it. For this, right click our project main folder and follow the path Deploy > Deploy to local Hybrid App Toolkit. This will deploy the application to HAT and it will create and compile our hybrid app.

13.deploy_HAT.png

14.HYBRID_APP_DEPLOYED.png

Now, we can run our project by right clicking our index.html > Run on > Android device.

15.RUN_ON_ANDROID_DEVICE.png

This will install our project’s application on our device. We now can click on the microphone button and see it in action (for the cordova plugin to work as the full Android speech recognition we will need to have an active internet connection).

onDevice.png

Debbuging our application

As a last step, we could debbug our application by using the chrome://inspect functionality. For this, we could follow the steps included on the following document: Getting Started with Kapsel – Part 3 — AppUpdate (SP09+

17.DEBUGGING_2.png

16.DEBUGGING.png

Conclusion

I hope you have enjoyed this blog, I see a lot of really interesting tools and concepts that SAP is providing us, and that we can apply trying to build better experiences for our users. I hope that you find this control useful too and that you may take this idea and implement this functionality in your projects. Thanks a lot ! Happy coding 🙂

Note: You can get the full source project here: eamarce/SAPUI5CustomControlApp · GitHub

Thanks!


Assigned Tags

      13 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Former Member
      Former Member

      Great tool and great explanation! Improvement at it´s finest!

      Author's profile photo Former Member
      Former Member

      Hi,

      I was able to run the application on the web successfully, but when I tried it on the android device, it shows a blank page.

      just for clarification, on the web, the microphone button is disabled, it must be disabled itself on the web application right?

      Author's profile photo EDGAR MARTINEZ
      EDGAR MARTINEZ
      Blog Post Author

      Hello Sanjo,

      That's correct, on the web the mic button would be disabled due to validations to assure the application is a cordova app.

      Regarding the blank page it may be the case that a component is not in place, were you able to add the SpeechRecognition plugin to the project?

      Best Regards!

      Edgar Martínez.

      Author's profile photo Former Member
      Former Member

      Hi Edgar,

      Yes, I was able to add the SpeechRecognition plugin correctly. Can you help me how to debug the app that I have installed on the phone? I tried to inspect via chrome, but my app isn't detected on the webview.

      Thanks,

      Sanjo Thomas

      Author's profile photo EDGAR MARTINEZ
      EDGAR MARTINEZ
      Blog Post Author

      Hi Sanjo,

      Sure, We can think about something to debug it... Perhaps you could send me your current .APK file and I could give it a try inside the Chrome Inspector.

      If you wish we can continue this conversation by direct messaging.

      Best Regards!

      Edgar Martínez.

      Author's profile photo Former Member
      Former Member

      Hi Edgar, my issue was resolved. After debugging, i found line 29 in speechrecognitioninputcontrol.js wasn't working. I just commented the code and it worked.

      The issue was with this code: var isCompanionApp = sap.hybrid.getUrlParameterName("companionbuster");


      Author's profile photo EDGAR MARTINEZ
      EDGAR MARTINEZ
      Blog Post Author

      Hello Sanjo,

      Thanks for your input and your interest on this content. I will be working a bit on the code to make that part a bit more flexible and then release a new revision,

      Thank you!

      Best regards!

      Edgar.

      Author's profile photo Swati Garg
      Swati Garg

      Good one. Thanks for sharing.

      Author's profile photo Albert Abdullin
      Albert Abdullin

      It's very useful and helpful blog with detailed informatoin. Thank you very much!

      I created App in newer environment. Ant it runs on my Samsung Tab. There is one issue that I 'm trying to resolve. Could you please tell who currently can help to find out the reason or give me the hint? In chrome://inspect I see that SpeechRecognition objects are in place but the button still stays inactive.

      Thank you very much in advance!

       

      Author's profile photo Albert Abdullin
      Albert Abdullin

      Tthe actual reason was that the App was detected as "sap.hybrid == undefined".

      Author's profile photo Hao Dai
      Hao Dai

      Hi Albert,

       

      I faced the same issue with yours, could you pls share your solution kindly? Thanks.

      Author's profile photo Hao Dai
      Hao Dai

       

      Thanks Edgar! It works in my Android device now.

       

      I list something which is different with the blog when I tried to build this App.

       

      1. I used the SAPUI5 Mobile Kapsel Starter Application(Deprecated) template when creating the project, because the WebIDE does not have New > Hybrid Mobile Project from Web App.
      2. I commented all "sap.hybrid*" related statements in js files.
      3. I can run the App in Android 4.4.2, but failed in Android 5 and 7.

      Just for your information.

      Thank Edgar, this blog is so great, teaching me a lot of things.

       

      Thanks &  Best Regards,

      Hao Dai

      Author's profile photo EDGAR MARTINEZ
      EDGAR MARTINEZ
      Blog Post Author

      Hello Everyone,

      Thank you for your comments, I'm glad to hear you have found this useful.

      I will try to include some of these comments and considerations as changes for a re-edition of the blog.

      Thanks a lot!

      Best Regards!

      Edgar Martinez.