Voice over control in SAP Fiori

Agenda:

To provide voice based search and navigation support to standard fiori applications running on browsers.

Currently as we all know, we can search for a fiori application in fiori Launchpad using the search toolbar. However we use text as an input for searching the application. To provide an enhancing feature in searching a fiori application, I have used voice command based search for fiori application here.

Target Audience: Code developers with UI5 and browser related application development skills.

Step 1:  Our fiori Launchpad application will be in /UI2/USHELL as a BSP Application. Therefore we need to open SE80 t-code to open the application.

Step 2: Open your FioriLaunchpad.html file under a Page Fragment>shells>abap where our Launchpad File is placed.

Step 3: Add the below code in your fiorilaunchpad.html file.

// clean fiori loading screen markup before placing main content
    var oContent = sap.ushell.Container.createRenderer("fiori2");
    var renderersExt = sap.ushell.renderers.fiori2.RendererExtensions;
    renderersExt.addHeaderEndItem(new sap.ushell.ui.shell.ShellHeadItem('clickToSpeak', {
      icon: 'sap-icon://microphone',
      tooltip:'Click to speak',
      showSeparator: false,
      press: onMicPress
    }));

Context of the above code:

• Variable oContent is the shell container created where the fiori application runs (this is generated by SAP).
• Now we need to extend the shell container, for that we create a variable renderersExt which contains the fiori shell extended libraries.
• Then, we are adding a button for the header item of the shell container with microphone as icon. Screen will look like below:

Following are the quick action performed inside the code when mic is pressed.

1. When the user clicks the mic button start service of the webkitSpeechRecognition API will be triggered using start() method which is available within the API.
2. We provide a 5 seconds time delay for the user to speak. Assuming 5 seconds is enough to speak an application name.
3. Once the speech is over, Stop service need to be triggered using stop() method in order to stop the API.
4. Then, the result will be available in onResult(event) method, where the event parameter holds the converted speech to text.
5. Now we will take the text input and set it to the search field of fiori Launchpad. The system does not provide any suggestion from search field now, since the text is set to search field and not typed.
6. So to get the suggestion for the list of fiori application in the fiori Launchpad, fire live change event need to be triggered for the search field in fiori Launchpad.
7. The first in the list will be identified and the first item will be fire pressed in order to get navigated to respective application.

Logic/Code behind speech recognition:

function onMicPress(oEvent){
      if('webkitSpeechRecognition' in window){
      var searchButton = sap.ui.getCore().getElementById("clickToSpeak").getParent().getAggregation("headEndItems")[0];
      searchButton.firePress();
        var speechRecognizer = new webkitSpeechRecognition();
        speechRecognizer.continuous = true;
        speechRecognizer.interimResults = true;
        speechRecognizer.lang = 'en-IN';
        speechRecognizer.start();
        jQuery.sap.delayedCall(5000, this, function () { // this has to be implemented so as the control comes back after 5 seconds
        speechRecognizer.stop();
        });
        var finalTranscripts = '';
        var oEvent = oEvent;
        speechRecognizer.onresult = function(event){
        var interimTranscripts = '';
        for(var i = event.resultIndex; i < event.results.length; i++){
        var transcript = event.results[i][0].transcript;
        transcript.replace("\n", "<br>");
        if(event.results[i].isFinal){
        finalTranscripts += transcript;
        }else{
          interimTranscripts += transcript;
          }
        }
        var shellInput = sap.ui.getCore().getElementById("searchInputShell");
        shellInput.setValue(finalTranscripts);
        shellInput.focus();
        sap.ui.getCore().getElementById("searchInputShell").fireLiveChange({finalTranscripts});
        jQuery.sap.delayedCall(2000, this, function () {
        var firstSuggestionItem = sap.ui.getCore().getElementById("searchInputShell")._oList.getAggregation("items")[0];
        sap.ui.getCore().getElementById("searchInputShell").fireSuggestionItemSelected({selectedRow:firstSuggestionItem});
        });
        };
        speechRecognizer.onerror = function (event) {
        };

Conclusion:

This is a very simple use case of taking voice as an input, and this use case can be expanded to various other scenarios like talk back feature and integration with other various voice engines like Google voice etc.,

E.g. voice commands:

• open timesheet
• get sales order 10851
• get purchase order 12004
• show open sale orders

I tried running this application in goggle chrome browser, and it worked. Dear techies, suggesting you to try this on other popular browsers and kindly share your feedback.

This is the part of overall fiori enrichment article written by Arun Santhanam. You can click here to know more details about the article.