Voice over control in SAP Fiori

[former_member245490]

Agenda:

To provide voice based search and navigation support to standard fiori applications running on browsers.

Currently as we all know, we can search for a fiori application in fiori Launchpad using the search toolbar. However we use text as an input for searching the application. To provide an enhancing feature in searching a fiori application, I have used voice command based search for fiori application here.

Target Audience: Code developers with UI5 and browser related application development skills.

Step 1:  Our fiori Launchpad application will be in /UI2/USHELL as a BSP Application. Therefore we need to open SE80 t-code to open the application.



Step 2: Open your FioriLaunchpad.html file under a Page Fragment>shells>abap where our Launchpad File is placed.



Step 3: Add the below code in your fiorilaunchpad.html file.

// clean fiori loading screen markup before placing main content

    var oContent = sap.ushell.Container.createRenderer("fiori2");

    var renderersExt = sap.ushell.renderers.fiori2.RendererExtensions;

    renderersExt.addHeaderEndItem(new sap.ushell.ui.shell.ShellHeadItem('clickToSpeak', {

      icon: 'sap-icon://microphone',

      tooltip:'Click to speak',

      showSeparator: false,

      press: onMicPress

    }));

Context of the above code:

• Variable oContent is the shell container created where the fiori application runs (this is generated by SAP).


• Now we need to extend the shell container, for that we create a variable renderersExt which contains the fiori shell extended libraries.


• Then, we are adding a button for the header item of the shell container with microphone as icon. Screen will look like below:

Following are the quick action performed inside the code when mic is pressed.

1. When the user clicks the mic button start service of the webkitSpeechRecognition API will be triggered using start() method which is available within the API.


2. We provide a 5 seconds time delay for the user to speak. Assuming 5 seconds is enough to speak an application name.


3. Once the speech is over, Stop service need to be triggered using stop() method in order to stop the API.


4. Then, the result will be available in onResult(event) method, where the event parameter holds the converted speech to text.


5. Now we will take the text input and set it to the search field of fiori Launchpad. The system does not provide any suggestion from search field now, since the text is set to search field and not typed.


6. So to get the suggestion for the list of fiori application in the fiori Launchpad, fire live change event need to be triggered for the search field in fiori Launchpad.


7. The first in the list will be identified and the first item will be fire pressed in order to get navigated to respective application.

Logic/Code behind speech recognition:

function onMicPress(oEvent){

      if('webkitSpeechRecognition' in window){

      var searchButton = sap.ui.getCore().getElementById("clickToSpeak").getParent().getAggregation("headEndItems")[0];

      searchButton.firePress();

        var speechRecognizer = new webkitSpeechRecognition();

        speechRecognizer.continuous = true;

        speechRecognizer.interimResults = true;

        speechRecognizer.lang = 'en-IN';

        speechRecognizer.start();

        jQuery.sap.delayedCall(5000, this, function () { // this has to be implemented so as the control comes back after 5 seconds

        speechRecognizer.stop();

        });

        var finalTranscripts = '';

        var oEvent = oEvent;

        speechRecognizer.onresult = function(event){

        var interimTranscripts = '';

        for(var i = event.resultIndex; i < event.results.length; i++){

        var transcript = event.results[i][0].transcript;

        transcript.replace("\n", "<br>");

        if(event.results[i].isFinal){

        finalTranscripts += transcript;

        }else{

          interimTranscripts += transcript;

          }

        }

        var shellInput = sap.ui.getCore().getElementById("searchInputShell");

        shellInput.setValue(finalTranscripts);

        shellInput.focus();

        sap.ui.getCore().getElementById("searchInputShell").fireLiveChange({finalTranscripts});

        jQuery.sap.delayedCall(2000, this, function () {

        var firstSuggestionItem = sap.ui.getCore().getElementById("searchInputShell")._oList.getAggregation("items")[0];

        sap.ui.getCore().getElementById("searchInputShell").fireSuggestionItemSelected({selectedRow:firstSuggestionItem});

        });

        };

        speechRecognizer.onerror = function (event) {

        };

Conclusion:

This is a very simple use case of taking voice as an input, and this use case can be expanded to various other scenarios like talk back feature and integration with other various voice engines like Google voice etc.,

E.g. voice commands:

• open timesheet


• get sales order 10851


• get purchase order 12004


• show open sale orders

I tried running this application in goggle chrome browser, and it worked. Dear techies, suggesting you to try this on other popular browsers and kindly share your feedback.

This is the part of overall fiori enrichment article written by arun.santhanam3. You can click here to know more details about the article.

[saurabh_vakil]

Hi ,

After speaking out the search term I am able to get reference to the search input on the launchpad (the ID in my case is searchFieldInShell). However, I get the error Uncaught TypeError: sap.ui.getCore(...).byId(...).setvalue is not a function on the below line of code and can't figure out how do I set the text to the search field on the launchpad.

shellInput.setValue(finalTranscripts);

Regards,

Saurabh

[former_member245490]

Hi Saurabh,

Thanks for reading the blog..

May i know, how have you taken the id searchFieldInShell.

1. Have you taken it by using developer tools --> Elements tab and find the Id by selecting the search input UI component (or)


2. Have you created a new search input and named the UI component with searchFieldInShell as Id.

In my case, I am using the first method to get the Id of the UI component. If you are using the second method please try to follow the method one and kindly let me know if you are able to achieve the same.

Thanks,

Magesh

[saurabh_vakil]

Hi Magesh,

Thanks for your response!

I too am using the first method for getting the ID of the search input similar to how you have described - Chrome developer tools -> Elements tab -> select the search input and get the ID.

I am working on a system with Fiori Front-end Server 3.0 (NW 7.51) and I have found out that instead of writing

shellInput.setValue(finalTranscripts)

I have to write

shellInput.input.setValue(finalTranscripts)

This way it worked fine for me.

Regards,

Saurabh

[former_member245490]

Thanks for the input !!

[saurabh_vakil]

Hi Magesh,

Have you tested this on mobile devices? I have tested it on Google Chrome on Android and iOS devices. Even though microphone access has been enabled for Chrome we find that on clicking the microphone icon on the launchpad header nothing happens.

Regards,

Saurabh

[former_member245490]

Hi Saurabh,

Yes, I have tested in android device with chrome browser(not yet tested in iOS device). It works fine for me. However, when i run it on Fiori client application in mobile it repeats the app name twice followed by no action taking place further. I am working on that now.

I would suggest you to refer this below link for microphone related issues.

https://productforums.google.com/forum/#!topic/chrome/FWhIZOag0k8

Thanks,

Magesh

[kammaje_cis]

Very cool.

Instead of directly editing the UI2/USHELL, I would suggest you create Fiori Plugin.

[Srikar]

Hi Magesh Kumar

Great Blog,

Do you have any sample UI5 Application without addding to the shell on the Fiori Launchpad.

This would be helpfull to me to understand the Speech Recognition API and the working flow of the application.

Do you have any chance of placing the application in GIT or JS-Bin.

[former_member245490]

Hi Krishna,

Thanks for your comments

[former_member245490]

Thanks Srikar,

Apologies for very late reply.

with few corrections in the code, you can implement the same solution in SAP UI5 application.

1. In step 3, Instead of creating a new ShellHeadItem(button) in header toolbar, you can create a button in UI5 application.


2. Under Logic/Code behind speech recognition section. You can create your input field and set the text to it, rather than capturing the text in "searchInputShell" (search toolbar in fiori launchpad).

Hope this helps!!

Thanks,

Magesh

[Former Member]

Hello Magesh,

 

Good Blog 🙂

Web Speech API has some limitation to browser. Are you aware of any solution / API which works in IE & Mobile(Safari).

https://caniuse.com/#feat=speech-recognition

Thanks for your support.

Sunil

[former_member245490]

Hi Sunil,

Thanks for your comment.

As mentioned in our parent blog , Voice recognition does not support in IE & Safari yet. So it is better to create a cordova application if there is a requirement for creating a browser specific application.

However, below are few popular voice to text search engines. Hope this would help!!

1. Artyom.js


2. Annyang


3. Voix JS


4. Mumble


5. Pocketsphinx.js


6. JuliusJS


7. voice-commands.js

Thanks

-Magesh

[govardan_raj]

hi Magesh Kumar,

Nice Blog !!!!!!!!

But currently on using webkitSpeechRecognition in development server we are getting error as "Network" , Kindly let me know how to resolve this issue.

 

Regards

Govardan

[former_member156553]

Hi Magesh Kumar,

 

how to write like  code in ABAP ADT, could you please help me.

 

best regards,

adi.

[mmmsaldua]

Hi Magesh,

 

Great blog. Question, have you tried if this is working in Fiori Client?

 

Thanks!

[mustafabelen]

Is 'webkitspeechrecognition' enable in Fiori Client. I couldn't make it work. Thank you.

[mustafabelen]

I figured it out. Just give permission for microphone to fiori client application.