Welcome to our third and final article of the blog series about chatbot methodology! Let’s get started with a brief recap on the 5 steps involved in a successful chatbot project.
The first step in a chatbot project requires you to build the Horizontal Coverage. In other words, building a dataset capable of understanding all questions entered by employees or customers to your bot.
It also provides you data-driven insights into what should be automated first. For more information about this concept, please read this first blog post.
The second step involves defining and implementing Use Cases for your chatbot that will perform actions for the user, using the SAP Conversational AI platform. You will find more details in this second blog post.
In this final article on chatbot methodology, we will cover the remaining phases that involve the critical aspect of engaging with your customers in testing the chatbot and driving adoption among customers.
The developed, tested, and trained bot is now ready for validation of conversational experience with your customers. This is accomplished in 2 phases – Alpha testing and Beta testing. Let us understand this better!
As mentioned before, Alpha testing is done in partnership with customers. It is therefore important to identify select customers who are able and willing to not only be your early adopters but also engage with you in testing the bot and sharing feedback.
Alpha testing of the bot is the first phase of testing with real customers to validate the conversational experience and identify areas of improvement.
The aim is to test both functional as well as non-functional requirements and ensure the readiness of bot.
It is important to test the bot in your own internal environment before testing in a client’s environment, so you can detect errors and fix them earlier before releasing the bot to customers.
As one of the key aspects of Alpha testing is for testing to be performed in your own lab, you will need to invite preselected users from client organizations to your lab and get them to test the chatbot in your environment. Here are the required steps:
Testing is expected to cover functionality (UATs and conversational flows), integration to channels as well as non-functional requirements (multi-language support, performance, etc.)
A detailed analysis of the feedback will help you identify the root causes and define actions for improvement. The next logical step is the fixing of the bugs/errors; the testing repeats until users detect no more issues, all known issues are fixed and the bot is ready.
Beta testing is the final stage of testing with real users in clients’ own test environment with a different set of end-users. The bot is tested by a preselected set of users from select customer organizations that are willing to partner as early adopters of your chatbot.
The users are trained, guided, onboard, and provided bot access for a defined period; users interact with the bot and record their feedback and conversational experience.
Beta testing covers aspects including but not limited to functionality (UATs and conversational flows), integration to channels, non-functional requirements (multi-language support, performance, etc).
Issues identified are fixed and tested again for readiness.
Beta testing is important in order to test the bot in clients’ environment by real users, so you can detect errors and fix them before widely releasing to customers at large scale.
A typical Beta testing phase with customers span 9-12 weeks as illustrated in the below example.
We begin with Customer 1. We onboard all their users, provide them with the necessary training to use the chatbot. We give them approximately 2 weeks of testing, followed by about a week of feedback.
We repeat the cycle and sequentially test the bot with the second and third customers after all identified issues and bugs from the previous customer are fixed.
Your chatbot is now well tested, including in clients’ environments, and ready for wider scale adoption among customers!
Although the chatbot is now ready for wider adoption and usage, we recommend you release in an incremental and phased manner in clients’ production environment, ramped by user groups and customer groups.
The reason we recommend an incremental addition of users and customers is to:
There are certain pre-requisites for ramp-up:
The ramp-up plan and rate of customer adoption would depend on case to case basis.
Here’s an example with this illustration:
Project teams typically tend to overlook the critical aspect of retaining a customer support team that is expected to analyze logs on a daily basis, test, and train the chatbot for continual improvement. Take that into account!
In reality, a chatbot project is never complete as it presents itself a valuable opportunity to continually improve, thereby enriching the conversational experience for users.
An essential and often overlooked aspect is the need to monitor the logs – this provides useful insights into how users are expressing their intentions and frequency thereof. Often, identifying the next best use case to augment is based on logs analysis.
The other aspect of log analysis is the annotation of expressions to intents, and the ability to correct and re-annotate expressions to the right intents, ensure correct entity tagging, thus continually training the bot for a truly enriching conversational experience.
This brings us to the end of our blog on project methodology! You are now ready to apply the SAP Conversational AI project methodology for your chatbot project.
We believe following the recommended methodology would greatly contribute to your project success as it is proven and based on experience and refined over time.
Do not hesitate to contact our team if you need to implement a chatbot in your business.
Feel free to ask your questions about any aspect of project methodology on SAP Answers or use the comment section below, our team will be there to help.
Part 1/3: Why integrating horizontal coverage when building chatbots is key
Part 2/3: How to design a perfect use case for your chatbot
TechEd session: Best Practices and Tips to Create Your Chatbot with SAP Conversational AI
The first step in a chatbot project requires you to build the Horizontal Coverage. In other words, building a dataset capable of understanding all questions entered by employees or customers to your bot.
It also provides you data-driven insights into what should be automated first. For more information about this concept, please read this first blog post.
The second step involves defining and implementing Use Cases for your chatbot that will perform actions for the user, using the SAP Conversational AI platform. You will find more details in this second blog post.
In this final article on chatbot methodology, we will cover the remaining phases that involve the critical aspect of engaging with your customers in testing the chatbot and driving adoption among customers.
The developed, tested, and trained bot is now ready for validation of conversational experience with your customers. This is accomplished in 2 phases – Alpha testing and Beta testing. Let us understand this better!
Step 3/5 – Alpha Testing
As mentioned before, Alpha testing is done in partnership with customers. It is therefore important to identify select customers who are able and willing to not only be your early adopters but also engage with you in testing the bot and sharing feedback.
What is Alpha testing?
Alpha testing of the bot is the first phase of testing with real customers to validate the conversational experience and identify areas of improvement.
The aim is to test both functional as well as non-functional requirements and ensure the readiness of bot.
Why is Alpha testing important?
It is important to test the bot in your own internal environment before testing in a client’s environment, so you can detect errors and fix them earlier before releasing the bot to customers.
What are the required steps to conduct an Alpha testing?
As one of the key aspects of Alpha testing is for testing to be performed in your own lab, you will need to invite preselected users from client organizations to your lab and get them to test the chatbot in your environment. Here are the required steps:
- Preselected users are trained, guided, onboarded, and provided access to the bot.
- They are then invited to interact with the bot, test using UATs, and record their feedback and conversational experience.
- Users would then share feedback on what is working well, with emphasis on what is not working well that is potentially contributing to a lesser than desired level of conversational experience.
Testing is expected to cover functionality (UATs and conversational flows), integration to channels as well as non-functional requirements (multi-language support, performance, etc.)
What is the outcome?
A detailed analysis of the feedback will help you identify the root causes and define actions for improvement. The next logical step is the fixing of the bugs/errors; the testing repeats until users detect no more issues, all known issues are fixed and the bot is ready.
Step 4/5 – Beta Testing
What is Beta testing?
Beta testing is the final stage of testing with real users in clients’ own test environment with a different set of end-users. The bot is tested by a preselected set of users from select customer organizations that are willing to partner as early adopters of your chatbot.
The users are trained, guided, onboard, and provided bot access for a defined period; users interact with the bot and record their feedback and conversational experience.
Beta testing covers aspects including but not limited to functionality (UATs and conversational flows), integration to channels, non-functional requirements (multi-language support, performance, etc).
Issues identified are fixed and tested again for readiness.
Why is Beta testing important?
Beta testing is important in order to test the bot in clients’ environment by real users, so you can detect errors and fix them before widely releasing to customers at large scale.
What are the required steps to conduct a Beta testing?
A typical Beta testing phase with customers span 9-12 weeks as illustrated in the below example.
We begin with Customer 1. We onboard all their users, provide them with the necessary training to use the chatbot. We give them approximately 2 weeks of testing, followed by about a week of feedback.
We repeat the cycle and sequentially test the bot with the second and third customers after all identified issues and bugs from the previous customer are fixed.
Beta Testing Phase
What is the outcome?
Your chatbot is now well tested, including in clients’ environments, and ready for wider scale adoption among customers!
Step 5/5 – Ramp up and production
Although the chatbot is now ready for wider adoption and usage, we recommend you release in an incremental and phased manner in clients’ production environment, ramped by user groups and customer groups.
The reason we recommend an incremental addition of users and customers is to:
- allow effective monitoring and control of your customer experience
- bring in remedial and timely intervention as necessary
There are certain pre-requisites for ramp-up:
- Training and onboarding of users and transfer of knowledge with clients’ technical teams; also, readiness to accompany clients’ journey from go-live to business-as-usual
- Provisioning in an automated manner, to address the scale
- Having a dedicated customer support team to monitor logs, train, track and resolve issues
The ramp-up plan and rate of customer adoption would depend on case to case basis.
Here’s an example with this illustration:
Ramp up Plan
Project teams typically tend to overlook the critical aspect of retaining a customer support team that is expected to analyze logs on a daily basis, test, and train the chatbot for continual improvement. Take that into account!
Launching phases in one table
Continuous improvement
In reality, a chatbot project is never complete as it presents itself a valuable opportunity to continually improve, thereby enriching the conversational experience for users.
An essential and often overlooked aspect is the need to monitor the logs – this provides useful insights into how users are expressing their intentions and frequency thereof. Often, identifying the next best use case to augment is based on logs analysis.
The other aspect of log analysis is the annotation of expressions to intents, and the ability to correct and re-annotate expressions to the right intents, ensure correct entity tagging, thus continually training the bot for a truly enriching conversational experience.
Congratulations!
This brings us to the end of our blog on project methodology! You are now ready to apply the SAP Conversational AI project methodology for your chatbot project.
We believe following the recommended methodology would greatly contribute to your project success as it is proven and based on experience and refined over time.
Do not hesitate to contact our team if you need to implement a chatbot in your business.
Feel free to ask your questions about any aspect of project methodology on SAP Answers or use the comment section below, our team will be there to help.
Part 1/3: Why integrating horizontal coverage when building chatbots is key
Part 2/3: How to design a perfect use case for your chatbot
TechEd session: Best Practices and Tips to Create Your Chatbot with SAP Conversational AI
- SAP Managed Tags:
