Challenge Submission: SAP Conversational AI Chatbot helps to test SAP projects using Int4 IFTT

This blog is part of the SAP Conversational AI Tutorial Challenge 2021. As we work helping our customers to test their integrated SAP solutions effectively with Int4 IFTT interface testing automation solution, we see here an amazing opportunity to improve their user experience and thus overall testing efficiency. I’d like to share with You how our customers can use SAP Conversational AI to streamline their testing during regression and development phases of S/4HANA projects.

 

Our Case

 

Int4 IFTT enables automated testing of various integration solutions (SAP PO and CPI leading) and SAP back-end interfaces, that is SAP AIF, IDOC, ABAP proxies and others. Thanks to our robotic crawler, for selected scenarios we can easily automate test case creation, utilizing past successful messages as test case data. Given some basic configuration, our customers can thus quickly create tens of thousands of integration test cases, reflected in automatically structured test catalog. On top, there are test cases created by hand for the more complex scenarios, including back-end document validation. All together, it’s easy to have a broad test catalog with too many test cases to easily navigate by hand. Moreover, as Int4 IFTT is a ABAP based SAP Addon, there is certain proficiency needed with how to use SAP tools, that is not widespread among the often diverse general purpose testing teams in large organizations. 

 

Opening the tool up to more users thanks to its exposed API and features offered by conversational AI seems a natural way to evolve the tool and increase its adoption and finally customer success.

 

The chatbox in action:

 

Technical architecture

Int4 IFTT exposes dedicated API’s to enable integration with other popular testing tools on the market. This allows transactional, REST and SOAP based calls to execute tests and get the results. We exposed these API’s through cloud connector to enable access to our private cloud and we set up the Conversational AI to consume these API’s, thus allowing a simple endpoint to execute tests even in a complex SAP landscape.

 

Figure – high level solution

 

How did we build int4 IFTT Bot? 

 

Bot creation

 

The first steps of bot creation are quite simple and standard. Go to Log In And Build Your Bot | SAP Conversational AI (https://cai.tools.sap) webpage and register to start working with the solution.

Your bot is going to perform different actions so the “Perform Actions” option is the correct choice. When it comes to predefined skills you only need “Greetings”, as all others will be custom. So far, your screen should look like the one shown below.

 

Figure – basic options for bot creation

 

In the next step you need to give your bot a name (in our case it’s IFTT), provide a simple description of the bot, choose its topics and a default language of the bot – in our case it’s English.

 

Figure – Bot details

 

As the last part you need to decide about the policy for the data that your bot uses and stores.

In our case it’s non-personal data which we want to store, to be able to enhance bot’s performance in the future. Also, our bot is intended for non-vulnerable users.

Bot visibility is the last option you need to decide about. In our case it’s “Private”

 

Figure – bot policy

 

After entering all the information you’re ready to press the “Create a bot” button and create your first CAI bot.

 

Entities

 

Your IFTT bot will need one custom entity, which will be responsible for retrieving a test case ID from the messages sent by the user. To create one you need to press the “+ Create” button in the “Entities” tab of your bot’s “Train” section.

Figure – creation of custom entity

 

Enter the name of the entity (“test-case-id” in our case) and select “Regex entity” as type. This way you will be able to provide a regular expression, which will be responsible for finding a test case ID in user’s messages.

 

Figure – details of custom entity

 

Click on the entry you just created and enter the regulars expression in the details of your entity.

Since the IFTT test case ID is a number consisting of up to 10 digits, the regex will be quite simple – “[\d]{1,10}”.

Figure – entity details

 

Intents

As mentioned above, you will also need custom intents for the bot to work properly. The standard intents, derived from the “Greetings” predefined skill selected during bot creation, are “@greetings” and “@goodbye”. These are responsible for saying “Hello” and “Goodbye” to the user. The custom intents will be responsible for the actual work that your IFTT bot does.

These are:

• @list-test-cases – responsible for listing all available test cases
• @run-test-case – responsible for executing given test case
• @test-case-results – responsible for showing results of the last run of the test case.

 

In the end the list of intents for your bot should look like shown below.

 

Figure – Intents

 

Each intent requires example expressions for which it should react. The more expressions you can provide, the more precise the reactions of the bot for user messages will be. 

Let’s create one intent together, the rest you can create using the same flow.

 

Enter the desired name of the intent into the input box (in our case it’s test-case-results) and press the “+ Create” button. Enter the description of the intent (Show test case results) and set the matching strictness to desired value (95 in our case).

The bigger the number the more certain AI has to be, to return the intent as found. Otherwise a fallback intent will be called.

 

Figure – Details of the @test-case-results intent

 

Now click on the row of your newly added intent to show its details. Here you are able to provide expressions which will guide the bot to this particular intent.

As @test-case-results is strictly connected with a test case ID (the results should be shown for a particular test case), the example phrases should also contain an example test case ID. Some examples may be:

• 18810 results
• Show 18810 results
• Display current results for 18810
• The result of 18810

 

Again, make as many expressions as you can think of, as this will greatly improve bot’s reactions. As you can see from the image below, in our case it’s 52 expressions. You can also notice that the #TEST-CASE-ID entity was found and assigned to the expression. This should be the case in all of your example expressions as this will guarantee that a test case ID is provided to the bot.

 

Figure – Expressions for test-case-results intent

 

Follow the same steps and create other intents (@list-test-cases and @run-test-case). Keep in mind, that @list-test-cases does not require a test case ID.

After you’re done with intents creation you need to train your bot. Press the “Train” button located in the top right corner of the administration panel.

List-Test-Cases skill

 

Skill is what enables a bot to do an action. You can create a skill by going into the “Build” section and pressing the “+ Add skill” button. In the window enter the skill name (list-test-cases), select “Business” as type, enter a title (List Test Cases) and press “Add”.

 

Figure – Creating the list-test-cases skill

 

You should now have a new skill in your “Build” section and the canvas should look something like the image shown below.

 

Figure – Build section after adding the list-test-cases skill

 

Click on the name of the newly created skill and go to the “Triggers” section. Press the input box located next to the “IF” statement and select the @list-test-cases intent. Press the “Save” button and choose “is-present” as a condition. This will ensure that the list-test-cases skill is triggered only if the @list-test-cases intent is found.

 

Figure – Triggers section of list-test-cases skill

 

Now go to the “Actions” section and press the “Add new message group” button. Select the “Connect external service” and choose the “Consume API service” option. Select “GET” as the method and enter your IFTT REST API Test Cases endpoint address. Provide any additional filters, if required. Select the “Basic authentication” as the authentication method and provide your username and password.

 

Figure – IFTT REST API call in the actions section of list-test-cases skill

 

Switch to the “Response” section of the message and enter “test_cases” in the input field on the left. This will ensure that the data returned from the API is stored under the test_cases object in the returned JSON.

 

Figure – Response settings for IFTT REST API call in the actions section of list-test-cases skill

 

Press the “Add new message group”, then the “Send message” button and choose “Custom”, in the input on the left choose “List” as message type and paste the code listed below as the message’s body. This will result in a list of all test cases returned by the IFTT REST API endpoint. The list will have a test case description, ID and last run result information.

 

{

  “type”: “list”,

  “delay”: “0”,

  “content”: {

    “title”: “Below is the list of all available test cases. Please pick one.”,

    “subtitle”: “”,

    “imageUrl”: “”,

    “total”: “{{length api_service_response.test_cases.body.DATA}}”,

    “upperBoundText”: “Too many options to display”,

    “buttons”: [],

    “elements”: [

       {{#eachJoin api_service_response.test_cases.body.DATA}}

      {

        “title”: “{{DESCR}}”,

        “subtitle”: “Created on {{CREATED_ON}}”,

        “imageUrl”: “”,

        “status”: “{{LAST_STATUS_TXT}}”,

        

        {{#compare LAST_STATUS “===” “S”}}

            “statusState”: “success”

        {{else}}

            {{#compare LAST_STATUS “===” “W”}}

                “statusState”: “warning”

            {{else}}

                “statusState”: “error”

            {{/compare}}

        {{/compare}},

        

        “description”: “#{{multiply CASEID 1}}”,

        “buttons”: [ ]

      }

      {{/eachJoin}}

    ]

  }

}

 

Figure  – Test cases list definition

 

After pressing save you will have your first skill ready. Your bot should now react to any of the expressions provided in the @list-test-cases intent. 

 

Please note: after each API call there should be a handler for API result (status_code in JSON). This blog does not go into error handling details.

 

Run-Test-Case skill

In the “Build” section of your bot press the “+ Add skill” button again. Provide a name of the skill (run-test-case), select “Business” as type and input a tille (Run Test Case).

Find the newly created skill on the canvas and click on its title.

 

Go to the “Triggers” section. Select the @run-test-case intent and select “is-present” as a condition. Next, press on the “+” button on the right hand side of the newly created condition, select the #TEST-CASE-ID entity and select “is-present” as a condition. Make sure there is an “AND” word between the two conditions.

 

Figure Conditions in the “Triggers” section of the run-test-case skill

 

Go to the “Actions” section and press the “Add new message group” button. Select “Connect external service” and “Consume API service” option. Select “GET” as the method and enter your IFTT REST API Test Runs endpoint address. Select the “Basic authentication” as the authentication method and provide your username and password.

 

Figure – IFTT REST API GET call in the actions section of run-test-case skill

 

Go to the “Headers” section and add a custom header for the CSRF token. Put “fetch” in the value field.

 

Figure – Headers of IFTT REST API GET call in the actions section of run-test-case skill

 

Put “tc_token” in the value field in the “Response” section of the API call. Mark the “Include headers” options. This way all header of the response will be included in the JSON.

 

Press the “Add new message group” button again and select “Connect external service” and “Consume API service” options. This time select “POST” as the method and enter your IFTT REST API Test Runs endpoint address. Select the “Basic authentication” as the authentication method and provide your username and password.

 

Figure – IFTT REST API POST call in the actions section of run-test-case skill

 

Go to the “Headers” section and enter headers with values as shown below:

x-csrf-token – {{api_service_response.tc_token.headers.x-csrf-token}}
cookie – {{join api_service_response.tc_token.headers.set-cookie ‘;’}}

 

Figure – Headers of IFTT REST API POST call in the actions section of run-test-case skill

 

Go to the “Body” section and enter the text shown below as the body template.

{

    “DATA” : {

        “TEST_CASES” : [ “{{#test-case-id.value}}” ]

    }

}

 

In the “Response” section of the API call put “tc_run” in the input field. 

 

Now press the “Add new message group” button again, choose the “Send message” option and select the “Text” message. In the body of the message put text shown below:

Test case {{#test-case-id.value}}) executed successfully.

{{#if api_service_response.tc_run.body.DATA.[0].TC_PASSED}}Test case **passed**.{{else}}Test case **failed**.{{/if}}

 

Mark the “Enable Markdown syntax” option as selected.

 

Figure – Result message for the IFTT REST API POST call of run-test-case skill

 

Test-Case-Results skill

In the “Build” section of your bot press the “+ Add skill” button again. Provide a name of the skill (test-case-results), select “Business” as type and input a title (Test Case Results).

Find the newly created skill on the canvas and click on its title.

 

Go to the “Triggers” section. Select the @test-case-results intent and select “is-present” as a condition. Next, press on the “+” button on the right hand side of the newly created condition, select the #TEST-CASE-ID entity and select “is-present” as a condition. Make sure there is an “AND” word between the two conditions.

 

Figure – Conditions in the “Triggers” section of the test-case-results skill

 

Go to the “Actions” section and press the “Add new message group” button. Select “Connect external service” and “Consume API service” option. Select “GET” as the method and enter your IFTT REST API Test Cases endpoint address. Select the “Basic authentication” as the authentication method and provide your username and password.

 

Figure – IFTT REST API call in the actions section of test-case-results skill

 

Switch to the “Response” section of the message and enter “tc_details” in the input field on the left. This will ensure that the data returned from the API is stored under the tc_details object in the returned JSON.

 

Press the “Add new message group”, then the “Send message” button and choose “Text”. In the body of the message paste the text shown below.

 

Details about last execution of test case {{#test-case-id.value}}:

result: **{{api_service_response.tc_details.body.DATA.[0].LAST_STATUS_TXT}}**

executed by: **{{api_service_response.tc_details.body.DATA.[0].LAST_STARTED_BY}}**

executed on: **{{api_service_response.tc_details.body.DATA.[0].LAST_STARTED_DATE}}**

executed at: **{{api_service_response.tc_details.body.DATA.[0].LAST_STARTED_TIME}}**.

 

Don’t forget to mark the “Enable Markdown syntax” as checked.

 

Fallback skill

 

Last but not least, the fallback skill, which is a standard skill available by default. It enables the bot to send a message to the user in case no intent is recognized, or no skill is triggered due to conditions. This way the bot can tell the user that he/she should rephrase the request.

 

Go to the “Build” section of your bot, find the fallback skill and press its title. In the “Actions” section edit the message and put text as shown below.

I’m sorry, I did not understand your request. Could you rephrase it, please?

 

Figure – The fallback skill’s message

 

Now your bot will send a meaningful message to the user, in case no other skill is executed.

 

Detailed solution architecture

 

Following the detailed instructions, it makes sense to present the detailed architecture of the solution involving all the components of the final solution.

 

Figure – architecture

 

And finally

 

We are amazed how simple and straightforward it was to make this happen. Basically in one day we came from nothing, to “hello world” to being able to ask for test cases and their status. Then day two was used to tweak the API to foster test execution, improve listing and overall experience.

Even if You don’t currently use IFTT, with this example You can quickly connect any REST based application to a conversational engine and thus enable applications and features, otherwise difficult to access, to a large population of users.

In our scenario one of our clients already uses a similar solution to trigger test documents creation using IFTT replay mechanism. And it turned out to work outside pure testing! It made developers’ lives easier with massive creation of business documents needed for unit testing and experimentation. It’s fantastic to see how opening up a new, simple and “informal” endpoint to a complex application enables people to improve their productivity.

 

This solution was prepared by Mateusz Adamus and Tomasz Mackowski you guys are rock stars!!!