SAPUI5 – Cordova speech recognition input control
What if we could help optimize data input as much as possible for our mobile users?
This was the thought I had when I decided trying to create a custom SAPUI5 control that could gather data from speech recognition and place it for further processing.
I have found we can make it possible by using a Cordova plugin, and by deploying our SAPUI5 application as a Hybrid application. On the case we would want to convert an already existing application and enable our control we can use the feature of Hybrid Mobile Project from Web App on the SAP Web IDE, although this feature has been marked as experimental on the SAP Web IDE Hybrid App Toolkit Add-on documentation for now.
This control will behave as a normal sap.m.input with the speech recognition button disabled on the case our application is not running on an Hybrid container.
Here is a short demo on how the final control looks:
So let’s move on and see how we can create and use this control.
We are going to need the following tools:
- SAP Web IDE 1.15
- Hybrid Application Toolkit 1.8.2
- A mobile device (in my case Android) for testing.
- On this case, our Android device should be connected
- Have the proper drivers installed
- Have the Developer mode enabled
- Have Android 4.4 and the Usb debugging mode enabled, if you wish to debug the project with the chrome://inspect functionality.
- Getting the macdonst/SpeechRecognitionPlugin · GitHub plugin and integrate it with HAT ( How to do this will be explained later )
Creating a SAPUI5 project to build and test the control
Ok, so first thing first, we will create a SAPUI5 project application in order to be able to write, build and test our control.
So, we should enter on the SAP Web IDE, once there we should select File>New>Project and then specify we want a SAPUI5 Application Project. We will make this project hybrid at a later stage, so we can use the Cordova plugin for voice recognition.
We shall name it CustomControlApp and proceed.
Next we can should enter our namespace ( custom.controls.demo ), select the view type as XML and we introduce our view name ( CustomControlView ).
After selecting Finish, we will see that the following project structure will be created for us:
Adding the control to our project
Once that we have our project generated, we will need to a folder named control, under the project root directory. Next we will create a new file inside this folder named SpeechRecognitionInputControl.js where we will place our custom control.
Now, we can include the following code on our SpeechRecognitionInputControl.js control file:
We will not visit the whole process of creating a custom control on SAPUI5 by now, but here are a couple of blogs that may help you jump start on this process:
Basically, what we are doing on this control is inheriting from a sap.m.Input control first. By that we will have the properties and methods of this control. Next, we are aggregating a sap.m.button control to fire the Speech recognition process.We also check that our application is running on a Hybrid container, and creating a new SpeechRecognition object to use the functionality of the Cordova plugin.
Ok, so next, we will add the following code to our CustomControlView, so we can use our custom control on our sample application:
From this code whe can see that we are registering our xml namespace xmlns:custom=”control” and later we use it to declare our <custom:SpeechRecognitionInputControl> control.
Next on our controller we should add the following code:
The important part that we can see is that we should make the location of our control known to SAPUI5. We do this by using the sap.ui.localResources(‘control’) directive. Next we should import our custom control class with jQuery.sap.require(“control.SpeechRecognitionInputControl”).
Of course, we can add more instances of our control programmaticaly, and for this we include the onButtonPress event handler.
After doing this steps we could test our project to verify that our custom control is being added and our project runs. To do so, we can right click on the index.html file > Run > Run as > Web application option. At this step we will not be able to use the Speech recognition functionality yet, but we will do it by adding some extra steps next.
Adding the Speech recognition plugin to Hybrid Application Toolkit
Now we should add our Custom plugin to our HAT installation. For this, I recommend we follow the steps shown on this blog from Simmaco Ferreiro: How to use custom Cordova plugins with SAP Web IDE and Hybrid Application Toolkit .
What we need to do is basically getting the macdonst/SpeechRecognitionPlugin · GitHub from git, on our custom plugin folder for HAT installation, by using the following command:
git clone https://github.com/macdonst/SpeechRecognitionPlugin.git
Then, when we proceed to install the HAT, we should specify the folder where we have our custom plugins and we should select our Speech recognition plugin next.
Once that we have completed the HAT installation and generation of the Companion App, we will proceed to convert our existing SAPUI5 application to an Hybrid Application project from within SAP Web IDE.
Making our SAPUI5 application an hybrid application
To complete this step, we should have enabled the Hybrid App Toolkit plugin on SAP Web IDE, and have our HAT installed and running. We can find steps to do this on the following document: How to use Hybrid Application Toolkit (HAT) and test the apps
Now, to convert our existing SAPUI5 application project to a Hybrid Mobile Project, we can right click our main folder project and follow the path New > Hybrid Mobile Project from Web App.
This is a one step wizard where we should specify which project should be converted.
Next, we need will configure the information of the application that we are going to deploy.
To do this, we should right click our project’s main folder and follow the path Project Settings > Device configuration.
We here should specify the App Name: CustomControlApp, the App ID: com.custom control, and the Version: 1.0.0.
We have to specify the platform where our application will run under the Platform section. In my case I selected Android.
Next, with our HAT running, we should right click on the Custom button, under the Plugin settings and select our SpeechRecognition plugin.
Deploying our application
Now it is time to deploy our application and testing it. For this, right click our project main folder and follow the path Deploy > Deploy to local Hybrid App Toolkit. This will deploy the application to HAT and it will create and compile our hybrid app.
Now, we can run our project by right clicking our index.html > Run on > Android device.
This will install our project’s application on our device. We now can click on the microphone button and see it in action (for the cordova plugin to work as the full Android speech recognition we will need to have an active internet connection).
Debbuging our application
As a last step, we could debbug our application by using the chrome://inspect functionality. For this, we could follow the steps included on the following document: Getting Started with Kapsel – Part 3 — AppUpdate (SP09+
I hope you have enjoyed this blog, I see a lot of really interesting tools and concepts that SAP is providing us, and that we can apply trying to build better experiences for our users. I hope that you find this control useful too and that you may take this idea and implement this functionality in your projects. Thanks a lot ! Happy coding 🙂
Note: You can get the full source project here: eamarce/SAPUI5CustomControlApp · GitHub
Great tool and great explanation! Improvement at it´s finest!
I was able to run the application on the web successfully, but when I tried it on the android device, it shows a blank page.
just for clarification, on the web, the microphone button is disabled, it must be disabled itself on the web application right?
That's correct, on the web the mic button would be disabled due to validations to assure the application is a cordova app.
Regarding the blank page it may be the case that a component is not in place, were you able to add the SpeechRecognition plugin to the project?
Yes, I was able to add the SpeechRecognition plugin correctly. Can you help me how to debug the app that I have installed on the phone? I tried to inspect via chrome, but my app isn't detected on the webview.
Sure, We can think about something to debug it... Perhaps you could send me your current .APK file and I could give it a try inside the Chrome Inspector.
If you wish we can continue this conversation by direct messaging.
Hi Edgar, my issue was resolved. After debugging, i found line 29 in speechrecognitioninputcontrol.js wasn't working. I just commented the code and it worked.
The issue was with this code: var isCompanionApp = sap.hybrid.getUrlParameterName("companionbuster");
Thanks for your input and your interest on this content. I will be working a bit on the code to make that part a bit more flexible and then release a new revision,
Good one. Thanks for sharing.
It's very useful and helpful blog with detailed informatoin. Thank you very much!
I created App in newer environment. Ant it runs on my Samsung Tab. There is one issue that I 'm trying to resolve. Could you please tell who currently can help to find out the reason or give me the hint? In chrome://inspect I see that SpeechRecognition objects are in place but the button still stays inactive.
Thank you very much in advance!
Tthe actual reason was that the App was detected as "sap.hybrid == undefined".
I faced the same issue with yours, could you pls share your solution kindly? Thanks.
Thanks Edgar! It works in my Android device now.
I list something which is different with the blog when I tried to build this App.
Just for your information.
Thank Edgar, this blog is so great, teaching me a lot of things.
Thanks & Best Regards,
Thank you for your comments, I'm glad to hear you have found this useful.
I will try to include some of these comments and considerations as changes for a re-edition of the blog.
Thanks a lot!