Skip to Content

Agenda:

To provide voice based search and navigation support to standard fiori applications running on browsers.

Currently as we all know, we can search for a fiori application in fiori Launchpad using the search toolbar. However we use text as an input for searching the application. To provide an enhancing feature in searching a fiori application, I have used voice command based search for fiori application here.

Target Audience: Code developers with UI5 and browser related application development skills.

Step 1:  Our fiori Launchpad application will be in /UI2/USHELL as a BSP Application. Therefore we need to open SE80 t-code to open the application.

Step 2: Open your FioriLaunchpad.html file under a Page Fragment>shells>abap where our Launchpad File is placed.

Step 3: Add the below code in your fiorilaunchpad.html file.

// clean fiori loading screen markup before placing main content
    var oContent = sap.ushell.Container.createRenderer("fiori2");
    var renderersExt = sap.ushell.renderers.fiori2.RendererExtensions;
    renderersExt.addHeaderEndItem(new sap.ushell.ui.shell.ShellHeadItem('clickToSpeak', {
      icon: 'sap-icon://microphone',
      tooltip:'Click to speak',
      showSeparator: false,
      press: onMicPress
    }));

Context of the above code:

  • Variable oContent is the shell container created where the fiori application runs (this is generated by SAP).
  • Now we need to extend the shell container, for that we create a variable renderersExt which contains the fiori shell extended libraries.
  • Then, we are adding a button for the header item of the shell container with microphone as symbol. Screen will look like below:

Following are the quick action performed inside the code when mic is pressed.

  1. When the user clicks the mic button start service of the webkitSpeechRecognition API will be triggered using start() method which is available within the API.
  2. We provide a 5 seconds time delay for the user to speak. Assuming 5 seconds is enough to speak an application name.
  3. Once the speech is over, Stop service need to be triggered using stop() method in order to stop the API.
  4. Then, the result will be available in onResult(event) method, where the event parameter holds the converted speech to text.
  5. Now we will take the text input and set it to the search field of fiori Launchpad. The system does not provide any suggestion from search field now, since the text is set to search field and not typed.
  6. So to get the suggestion for the list of fiori application in the fiori Launchpad, fire live change event need to be triggered for the search field in fiori Launchpad.
  7. The first in the list will be identified and the first item will be fire pressed in order to get navigated to respective application.

Logic/Code behind speech recognition:

function onMicPress(oEvent){
      if('webkitSpeechRecognition' in window){
      var searchButton = sap.ui.getCore().getElementById("clickToSpeak").getParent().getAggregation("headEndItems")[0];
      searchButton.firePress();
        var speechRecognizer = new webkitSpeechRecognition();
        speechRecognizer.continuous = true;
        speechRecognizer.interimResults = true;
        speechRecognizer.lang = 'en-IN';
        speechRecognizer.start();
        jQuery.sap.delayedCall(5000, this, function () { // this has to be implemented so as the control comes back after 5 seconds
        speechRecognizer.stop();
        });
        var finalTranscripts = '';
        var oEvent = oEvent;
        speechRecognizer.onresult = function(event){
        var interimTranscripts = '';
        for(var i = event.resultIndex; i < event.results.length; i++){
        var transcript = event.results[i][0].transcript;
        transcript.replace("\n", "<br>");
        if(event.results[i].isFinal){
        finalTranscripts += transcript;
        }else{
          interimTranscripts += transcript;
          }
        }
        var shellInput = sap.ui.getCore().getElementById("searchInputShell");
        shellInput.setValue(finalTranscripts);
        shellInput.focus();
        sap.ui.getCore().getElementById("searchInputShell").fireLiveChange({finalTranscripts});
        jQuery.sap.delayedCall(2000, this, function () {
        var firstSuggestionItem = sap.ui.getCore().getElementById("searchInputShell")._oList.getAggregation("items")[0];
        sap.ui.getCore().getElementById("searchInputShell").fireSuggestionItemSelected({selectedRow:firstSuggestionItem});
        });
        };
        speechRecognizer.onerror = function (event) {
        };

Conclusion:

This is a very simple use case of taking voice as an input, and this use case can be expanded to various other scenarios like talk back feature and integration with other various voice engines like Google voice etc.,

E.g. voice commands:

  • open timesheet
  • get sales order 10851
  • get purchase order 12004
  • show open sale orders

I tried running this application in goggle chrome browser, and it worked. Dear techies, suggesting you to try this on other popular browsers and kindly share your feedback.

This is the part of overall fiori enrichment article written by Arun Santhanam. You can click on the below link to know more details about that.

To report this post you need to login first.

7 Comments

You must be Logged on to comment or reply to a post.

  1. saurabh vakil

    Hi ,

    After speaking out the search term I am able to get reference to the search input on the launchpad (the ID in my case is searchFieldInShell). However, I get the error Uncaught TypeError: sap.ui.getCore(…).byId(…).setvalue is not a function on the below line of code and can’t figure out how do I set the text to the search field on the launchpad.

    shellInput.setValue(finalTranscripts);

    Regards,

    Saurabh

    (1) 
    1. Magesh Kumar Post author

      Hi Saurabh,

      Thanks for reading the blog..

      May i know, how have you taken the id searchFieldInShell.

      1. Have you taken it by using developer tools –> Elements tab and find the Id by selecting the search input UI component (or)
      2. Have you created a new search input and named the UI component with searchFieldInShell as Id.

      In my case, I am using the first method to get the Id of the UI component. If you are using the second method please try to follow the method one and kindly let me know if you are able to achieve the same.

      Thanks,

      Magesh

      (1) 
      1. saurabh vakil

        Hi Magesh,

        Thanks for your response!

        I too am using the first method for getting the ID of the search input similar to how you have described – Chrome developer tools -> Elements tab -> select the search input and get the ID.

        I am working on a system with Fiori Front-end Server 3.0 (NW 7.51) and I have found out that instead of writing

        shellInput.setValue(finalTranscripts)

        I have to write

        shellInput.input.setValue(finalTranscripts)

        This way it worked fine for me.

        Regards,

        Saurabh

        (1) 
          1. saurabh vakil

            Hi Magesh,

            Have you tested this on mobile devices? I have tested it on Google Chrome on Android and iOS devices. Even though microphone access has been enabled for Chrome we find that on clicking the microphone icon on the launchpad header nothing happens.

            Regards,

            Saurabh

            (0) 
            1. Magesh Kumar Post author

              Hi Saurabh,

              Yes, I have tested in android device with chrome browser(not yet tested in iOS device). It works fine for me. However, when i run it on Fiori client application in mobile it repeats the app name twice followed by no action taking place further. I am working on that now.

              I would suggest you to refer this below link for microphone related issues.

              https://productforums.google.com/forum/#!topic/chrome/FWhIZOag0k8

              Thanks,

              Magesh

              (0) 

Leave a Reply