Improve Conversation Quality by Testing Your Chatbots with the New Test Tab

• Reduce the time needed for testing working scenarios for bots
• Detect and fix regressions quickly to assure quality

Let’s walkthrough on how to test your chatbot and ensure a consistent end-user experience.

NOTE: The Bot Testing feature is available only for the Enterprise Edition of SAP Conversational AI. It is currently not available for FAQ chatbots.

What you will need:

1. A chatbot with skills, intents, and entities that you want to test. Please follow this tutorial to build your first chatbot.
2. An account on SAP Conversational AI.

A bit about the bot

Today we are going to test a movie bot! Our movie bot takes in genre, language, and recording details from the user and suggests a movie.

Currently, we have a version ‘v1’ of the bot. Later, we will create a new version and add a movie theater suggestion functionality to it. We will then observe if the changes have affected our previous version ‘v1’.

NOTE: Check the Roles and Permissions section for read and write capabilities.

Step 1: Create your test document

First, we need to create a test document that is composed of test cases. A test case is an expression with its corresponding intent, entities, and skills.

Select your bot and go to the ‘TEST’ tab. On the Test Overview page, click the ‘+New’ button and enter your test document name. You will now see your test document created on the Overview page.

Step 2: Add expressions to test

Now we can add expressions to our test document. These expressions are the expected user inputs to your chatbot. Select the edit (pencil icon) besides the ‘Total test cases’ or click on the overflow menu (three dots) and select ‘Manage test cases’.

Click the ‘+New Expression’ button and enter an expression that you would like to test. You can now view the skill, intent, and entities that would be triggered when the user inputs the expression.

NOTE: Add multiple expressions to cover various scenarios the user might enter. The more expressions entered, the better your bot will be!

Step 3: Run your tests

It’s time to run some tests! Go back to the ‘Overview’ tab and click on the ‘RUN’ button against your document. Select the bot version ‘v1’ to run your tests against it and hit RUN!

Step 4: View your results

After a successful run, you will now see the test details within the results bar. A red bar indicates that some test cases have failed.  Also, you can see the number of test cases that have failed. If none of the test cases have failed, you will see a full green bar! Hooray!

Now, we wish to add new functionality to the bot. We will first create a new version ‘v2’ of the bot from the previous version ‘v1’. We add a skill ‘get-theater’ which will be triggered when the @get-theaters intent is detected.

Let us run our test document against ‘v2’.

We can see that two of the test cases have failed! Select the link indicating the number of failed results against your test document. You will be redirected to the test result details page on the ‘Document’ tab.

Here you can view the errors that might be due to a mismatched skill, intent, or an entity. On clicking the arrow on the right side of an expression, more details will be listed.

NOTE: You can see the details only if there are any failed tests. The details of successful test runs are not displayed.

NOTE: Add more test cases based on the results to improve the intents and entities in your bot.

To add more test cases to an existing test document, in the Overview page, click the overflow menu (three dots) and select ‘Manage Test Cases’.

If the run was unsuccessful, a red exclamation icon is displayed. You can click on it to view the error. You then need to re-run your test.

Step 5: Run multiple tests on your test document

After adding more test cases and updating the skills, intents, and entity set for the bot, you can start a new RUN against the bot version. To view all the test runs of a test document, you can select ‘View results’ from the overflow menu (three dots) on the right.

The list displays the test date, version, and results. To view the test run details, select the arrows on the right and you will be redirected to the details page.

Roles and permissions

With Read-only access, you may only view test cases and test runs.
With Read and Write access, you may also run, add, and delete test cases.

For a bot that belongs to a specific owner, a collaborator is assigned Read and Write permissions by default.

If your bot is a part of an organization, you will need the administrator to define the permissions as Read-only or Read and Write.

Currently, the permissions of the ‘Testing’ feature will be the same as the permissions set for the ‘Train’ feature of the bot.

What’s next?

We hope you now feel empowered to properly test your bot across various versions. We are continuously working on enhancing the bot testing experience to provide more flexibility and robustness with features in our roadmap. So, stay tuned for more and let us know your feedback!

For more information about our Bot Testing feature, please watch this video:

Ask your questions on SAP Answers or get started with SAP Conversational AI!