SAP Conversational Ai chatbot architecture and imp...

former_member61854 · ‎11-19-2019

The primary focus of this document is to discuss the various ways to implement SAP Conversational AI into your IT landscape while maintaining data privacy and security. This document will focus on scenarios where the user has data in an on-premise environment with varying degrees of data privacy constraints. Moving forward, we will refer to SAP Conversational Ai as SAP CAI.

Key topics being covered:

Why should you incorporate chatbots into your organization?

Key Concepts that you’ll need to understand for the remainder of the blog

High level overview of how CAI connects to your system

Implementation styles of Conversational Ai

Standard implementation – Cloud solution utilizing on premise database
- Data privacy with this approach
- Identity protection in this approach

SAP CAI NLP Integration – Least public cloud exposure, maximum data privacy
- Data privacy with this approach

SAP CAI hybrid Integration – zero exposure to back end data
- Data privacy with this approach

Why should you incorporate chatbots into your organization?

SAP Conversational Ai is an end to end solution which allows you to build, train, deploy and monitor artificial intelligent chatbots. Artificial intelligent chatbots are software that simulates human conversation.

The genius is making the complex simple and that is the purpose of Conversational Ai. As we move forward, our software applications and business processes become more complex for our employees and customers. Chatbots make navigating complex IT landscapes a breeze and that’s why, we at SAP are integrating SAP CAI into our own solutions.

Key Concepts that you’ll need to understand for the remainder of the blog:

Expressions: The sentence entered by the user is referred to as an expression.

Entity: a keyword that is extracted from an expression.

Intents: Intent is a box of expressions that mean the same thing but are constructed in different ways. Intents are the heart of your bot’s understanding. Each one of your intents represents an idea your bot

can understand. Here’s an example of classifying expressions into intents:

High level overview of how CAI connects to your system:

To better understand how CAI fits into your IT landscape, we need to start with the pieces involved:

Natural language processing engine (NLP Engine)

Bot Builder

Bot logic

Bot connector

The diagram below will help you understand how these pieces fit together. We will go into more detail further down the article.

NLP engine:

The natural language processing (NLP) engine is responsible for using artificial intelligence to better understand a user regardless of how the sentence is phrased. In the past, chatbots relied on a rule-based framework that required specific queries to provide results, it’s cumbersome, inefficient and didn’t scale well. The NLP engine extracts and returns actionable data including recognized intents, entities, user sentiments from expressions.

Bot Builder:

The bot builder, sometimes referred to as dialog runtime, is the graphical user interface(GUI) where you can build out the conversation flow. Simply put, this is where you would tell the bot how to respond once we know what the user wants. The bot builder provides an intuitive user experience with a drag and drop(no code) environment that accelerates the development process.

Bot logic:

The bot logic is a middle ware application that can be hosted on SAP Cloud platform. This application is responsible for:

calling and ingesting OData services from the back-end database/system and exposing that information to SAP CAI.

Using the bot logic, you can incorporate additional custom logic as an extension of the logic defined in the bot builder.

Bot logic can be written in the programming language of your choice and must be exposed as a web API.

Bot connector:

The bot connector is an adaptor which helps SAP CAI connect to various communication channels such as webchat, slack, Microsoft teams, etc. For a full list, refer to this link.

Important note: the bot connector can be entirely redeveloped on premise to fit custom requirements.

Implementation styles of Conversational Ai

Now that you understand the pieces that are involved, we are going to look at various implementation styles. Here are the key factors you should think about when choosing the style of implementation:

Are you currently using a private cloud solution, on premise solution or a public cloud?

Can you expose your chatbot conversations to the cloud?

Are you planning on making the move to the cloud?

Legend to help you understand the architectural diagram

Standard implementation – Cloud solution utilizing on premise database:

In this example, we are looking at the implementation with the least amount of development work required. This company:

Requires a backend on premise database for information

They would like to use SAP Cloud platform to host bot logic

Taking advantage of the prebuilt Bot connector to connect to various channels like skype, webchat, etc.

Now I’ll walk you through how the entire process end to end:

The user will type in an expression in one of the channels you see below

The expression enters the bot connector and gets translated into a format that SAP CAI can process.

Then the expression is pushed into the dialog engine(a combination of the natural language processing(NLP) engine and Dialog runtime)

The NLP engine extracts the intents and entities from the expression.

The dialog runtime controls the flow of the conversation based on the extracted intents and entities.

If the conversation requires information from the back-end system to move forward, the dialog engine from CAI will call the bot logic.

The bot logic will receive information as a JSON package.

this package will include the state of the conversation, confidence scores, triggered skills etc.

Using this information, the bot logic will connect to the backend system and retrieve the required information or trigger a certain transaction.

The bot logic is also responsible for formatting the response to the chatbot so CAI can understand the information. To reiterate, this is custom coded logic, so you have the freedom to customize the bots actions as much as you’d like to from the back end

To expose the data from your back-end system in a safe and secure manner, OData services can be exposed from your storage through services such as SAP Gateway. The cloud connector will allow you to expose these OData services without opening ports on your firewall. If your database is in the cloud, you will not have to worry about using a cloud connector, you can directly connect your bot logic to your data if you expose it as a web API.

Key points:

You will be leveraging the natively built Bot connector, NLP engine and Dialog runtime.

The bot logic is the only application you will have to develop on your own.

You will have to configure the SAP Gateway and SAP Cloud connector for this implementation style

Data privacy with this approach

Your data will be exposed to the cloud in this approach. Conversational Ai has data protection policies in built to ensure you comply with GDPR.

Identity protection in this approach

You can enable single sign-on for identity protection and to ensure people have access to only the information they are permitted to see. Here’s a great blog providing step by step instructions to enable SSO with Conversational Ai. Link to blog

SAP CAI NLP Integration – Least public cloud exposure, maximum data privacy

This approach leads to the least public cloud exposure and is primarily used for augmenting applications hosted in your intranet. The architecture can also ensure no sensitive data is exposed to the cloud. This would be ideal for a private cloud or on-premise customer that wants the least amount of cloud exposure. This approach requires more development effort as it uses less of the prebuilt content. To reiterate, is only suitable for internal use cases.

In this infrastructure, almost the entire chatbot ecosystem will remain within the client infrastructure whether that is on premise or a private cloud. Now let us dive into how this architecture works.

Webchat will be your on-premise channel for your users to communicate with your bot so the first step would be for you user to enter an expression x into webchat.

Once webchat receives an expression, it will route it to your on premise bot connector.

1. The bot connector(open source) must be redeveloped and deployed in your on-premise environment.
2. The bot connector is responsible for translating the user input into a format that can be consumed by SAP CAI.

The bot connector then transfers the expression x to the bot logic.

In order to extract the intents and entities within the expression x, the bot logic will then send the expression to the NLP engine on SAP CAI(Cloud). In order to maintain data privacy, you can first encrypt all your crucial information in expression x before sending it to the NLP engine. This way, none of your sensitive information will ever be exposed the cloud.

The NLP engine will send a response with the intents and entities, once the bot logic receives this, it can move the conversation forward with an adequate response which sometimes requires back end API calls to the on premise database or backend system.

Data privacy with this approach

This is entirely an on-premise setup where the only instance of the data leaving your firewall is in the form of an API request to NLP engine within CAI to extract the intents and entities.

Key points:

In this architecture, the bot connector and bot logic will be hosted on the on premise/private network

With this architecture, you can ensure no back-end database or back end system information is exposed to the cloud.

You can make sure all expressions being sent to the cloud have all their sensitive information encrypted before leaving your on-premise environment

SAP CAI hybrid Integration – zero exposure to back end data

This implementation style is perfect if your goal is to avoid exposing O Data services from your back end. The major differentiation is that you will host the bot logic on your on-premise system ensuring that your back-end data never has to leave the premise. Now let’s dive into how this would work:

The user enters the expression into one of the various channels (Webchat, slack, etc.) and is passed to the bot connector.

Bot connector which is hosted on SAP Cloud platform then sends the expression to the bot logic which is hosted in your on-premise network.

The bot logic handles the expression and makes a request to:
1. The backend system if that’s required to move the conversation forward.
2. The NLP engine in the cloud to extract entities and intents from the expression

Once this information is collected, the bot logic has all the information it needs to move the conversation forward.

Data privacy with this approach

In this approach, the information entered by the user is the only information that would be exposed to the cloud. Sensitive information within expressions entered by the user can be encrypted using the bot logic before sending it to the NLP engine. On top of that, all your back-end system data will stay within your firewall since the bot logic is being housed on premise.

Key Points:

Your back-end system data will not be exposed to the cloud.

You will need to custom build your bot logic as you are not using the inbuilt dialog runtime module of SAP conversational Ai.

This is ideal for a customer facing bot where you must provide confidential information that you don’t want to expose to SAP’s conversational Ai.