User Experience Insights
What are AI interaction patterns & why do we need them?
Communication target: The topic of this article concerns the set-up of a continuous series of small articles for https://community.sap.com. For more details on my motives and profile please also read my first article posting about approaching ‘intelligence’ as a designer.
Does SAP think about repeatable solutions to AI interaction problems?
SAP’s AI Design Practice team has investigated various AI products and examined different interaction patterns for human-AI-interaction in the past. Our hypothesis was that, even though the individual objective of applications might vary a lot, how their solutions surface on the UI will eventually consolidate within a recognizable pattern. Most commonly we can view such a pattern emergence in the quite popular introduction of recommendation systems in many of today’s products in the consumer, as well as B2B, markets.
What is an interaction pattern in UX?
‘Patterns‘ are a common theme in UX / interaction design. Without them, designers would need to figure out each time anew how UI elements can be manipulated and interacted with to achieve various user objectives. (Find an in depth definition for ‘interaction pattern’ here linked from IxDF.org)
If you are already familiar with the concept of UX pattern, you might want to skip to the next passage.
Thanks to all the prior work of the designers and UX researchers that came before us, many patterns in Human Computer Interaction (HCI) have been discovered and documented in detail already, so we are now able to choose from a rich set of options. For example, if we want to give the user the ability to revert wrong entries, we can make use of the “undo / redo” pattern. If we would like to guide our user in navigating our content, we use “progressive disclosure” to reveal information bit by bit. And if you would like to allow your user to change the order or location of items intuitively, the pattern of “drag & drop” provides a readymade solution.
Are UI and UX patterns the same?
The two terms are mostly used interchangeably, or confused with other things like style guides or component libraries. In short, a UX pattern is defined by the user’s interaction along their journey through the product and therefore includes the individual steps the user needs to take in order to reach their objective / execute a certain operation. UX also includes the UI elements through which a user executes those steps and how the system returns feedback to them about the success of their actions.
The user would like to change the position or location of a given item. There are multiple patterns offered to achieve this. The user might choose this in a classical way by selecting the item and then using a dedicated action to move it. However, a drag-&-drop feature provides an intuitive way to grab the item and just drag it to its new location, which requires less steps to be performed. In order to enable this feature, multiple UI components must work together. For example, once the item has been grabbed out of its position, the item list must reflect that the item has been removed. The mouse cursor has to visualize that its actually holding onto the item while the user keeps the left mouse button pressed / their finger down on the touchscreen. As a consequence / What ultimately constitutes the pattern is not a single UI element but the reciprocal interaction between a multitude of them. You probably get the point.
In summary, interaction patterns are a means to provide designers with systematic, standardized, and well-tried solutions. A UX-pattern is like a recipe that, when followed strictly, will give your users recognizable and reproducible experience. In contrast to that, ‘UI-patterns’ are constrained to the ability and behavior of a single UI element (e.g. a radio-button).
If you would like to further follow up on the differentiation of concepts in interface design, I suggest you get familiar with the “Atomic Design” hierarchy by Brad Frost as a memory hook.
Are there UX patterns in “Human AI Interaction” as well?
One prominent example of a repeatable AI interaction pattern is the goal to provide a ‘recommendation’ to the user, like a movie on Netflix, a product on Amazon, or a song on Spotify. The recommendations might look different in how they are displayed in the different applications, but how the user interacts with them mostly works and feels very similar.
Roughly described, you have several things you want to recommend to the user under a defined precondition, such as content, actions, or inputs. You can consolidate types of recommendations under defined objectives. For example, “movies similar to the titles you watched recently”. You also provide the user with the option to accept or reject an offer and store their decision as feedback to the recommendation system so machine learning can further refine future recommendations. But there are interaction scenarios and user–needs beyond recommendations in the interaction between AI & human that are not mentioned yet.
What might be the most common AI-UX pattern at SAP?
I try to name the patterns in a sequence here that align to the way we examine communication-needs between users and AI. Apart from that, the patterns do not have any inherent order or hierarchy. It also depends on the application scenario if you will encounter these patterns in isolation or in combination.
Predictions / Forecasting
Informing the user about the nature and origin of a given AI output / prediction — Managing the expectations of the user is the most important goal of UX for AI. We don’t want to leave the user with any doubts or ambiguity about what the source of this prediction or forecast is and how it needs to be treated. All AI or machine learning results are predictions, therefore have a remaining margin of error and need to be judged by an enabled user, so profound and informed decisions can be taken upon them.
Informing the user about a changed situation, important changed parameter, or forecasted event which bears potential implications on the objectives of the user and/or company.
Providing the user with a guiding pre-assessment on what the most likely or matching choice among all predicted possible solutions is, given a defined objective or set of criteria. A recommendation can only be formalized under the prerequisite of given desirable objectives.
Give the user access to the source of information, i.e. data, upon which the AI has based its prediction. Provide insight into the actual performance over time of the AI model. And where possible, identify the most influential data points with the highest impact, therefore informing the model’s mathematical logic behind its result.
These are the major patterns which we project to have the most tangible and powerful effect on the way users work with software. There may be other patterns, or subordinated variation of what we’ve listed. However, focusing on the most frequent AI use cases is where we can start innovating and gathering experience — and eventually we will discover new things on the way in the future.
An ‘AI UX pattern’ can be a tool to address the challenges and most common requirements of AI product design. They will help users to establish mental models and routines to deal with AI solutions, therefore reducing ambiguity and mental load on users around what results to expect and how to use AI infused applications.
The advantages of using patterns
• Your users do not need to re-learn how to use your software as they can recognize and use a pattern intuitively that is commonly and regularly used throughout the product, product portfolio or industry already.
• You do not need to reinvent new solutions for common interactions that are frequently needed and applicable to a variety of situations.
• Standardized interactions are less prone to human error and are generally faster to process cognitively.
Missing any pattern?
Our team is constantly looking to discover and explore novel aspects of human-AI-interaction and wants to learn about your product scenarios and how you have solved the challenges of delivering human-centered-AI solutions to your users.
Which patterns did you came across and how did you apply them? We would love to hear about you stories, struggles and successes, and invite you to share them with the community.
If you are interested in further exploring the subject of UX for intelligent systems, we invite you to read our other blog articles mentioned below as further readings, revisit our section in the SAP Fiori UX Design Guidelines or join the exchange here in the SAP Community. If you like this article, leave a comment and let us know about your experiences with UX matters in machine learning scenarios.
Please make a visit to the blog posts of my colleagues Joachim Sander or Thomas Reiss on the subject of design for intelligent systems and intelligent ERP.
Many thanks to my team mates of SAP’s AI Design Practice for feedback and cross reading.
Thanks for the overview. I am curious why you added Notifications as a AI pattern? The way I see it - it is a simple conditional event. Can you clarify?
Thank you for the feedback.
You are correct that classical notification are triggered through conditional rules & thresholds. That is where we mostly are today. Please read our post about Situation Handling too, which is a product that is exploring more "intelligent" ways of informing the user about business critical situations.
The same critique might apply to many recommendation systems which can as well be driven by a rule engine behind the scenes. It is not a definitive prerequisite that a recommendation must be created by a sophisticated AI/ML model.
The aspect which would turn the thing upside down is an quality of how it was created. If we think of the AI as a "generative" actor which creates the entire content or parts of the notification on its own without being explicitly thought / programmed how to do it (Still, this might by fiction as of today) – we might be surprised with the results and variety of messages the AI produces.
Already we can experience the power of generative AI's by the example of OpenAI projects like GPT-3 or chatGPT. Those are highly flexible, potent, and convincing applications at first sight, but at the same time produce also unpredictable, unexpected, and confident but inaccurate results. The user might find them convincing but he should not rely on them blindly (e.g. chatGPT can generating C# code based on the users description that works but is not "correct" in a experienced developers point of view).
Thought experiment: If someone would attempt to invent an intelligent system one day that would without explicit programming, gather, detect and discover conditional informations in the system, which might or might not reveal valuable insight to the user, and than bring its own conclusion to the users attention via notification so he might be able to act ahead of time – UX designer like me would like to make sure that the user is able to recognise the difference between a AI generated notification and a human configured one.
Thanks Alexander Schreiber for sharing your insights about the relationship between AI and UX, really interesting. Looking forward to the series!
Thanks Alexander, great article.
One question: how would you categorize the scenario when an AI service takes a decision for the user and then either informs the user of the decision taken or asks for the user validation?
e.g. I take a picture of a receipt and the AI service automatically creates a request to reimburse the expense for me. Then I am either just informed that the request was entered or the system asks me if the request is correct and if it should be sent for reimbursement
My first question here would be whether or not a AI system is accurate and reliable enough in a given use case to perform fully automated / autonomous actions within a system in the first place?
In most scenarios, the desire to fully automise certain tasks will remain wishful thinking for now. Especially as task complexity and stakes are increasing in those remaining business scenario which couldn't be automised via classical rule engines or simpler predictive models – it is not secured prematurely that a more sophisticated AI/Ml technique can do the job.
So you have correctly deduced that in such cases there needs to be some user involvement to validate and confirm the decision of the AI before it is actually applied in the system.
You might would like to follow up further investigating the topic of so called "Human-Intervention-Concepts", to name most prominently "Human-in-the-Loop" which touch on all of the described patters in my article to a certain degree.
The AI system is monitoring the flow of information and makes predictions based on the modelled scenario. If a given desirable or undesirable outcome is predicted and the AI is very confident about it, it should notify the user about it. This system may also be able to infer counter measures that have been successful in the past and present suggestions to the user which action(s) he might want to choose. It would be helpful for the user if the AI would be able to explain its suggestions too, indicating to the user which information let to the proposed recommendations. And at the end, leaving the final decision and control to the user and learning from the users feedback to get better and more accurate over time until it makes so few mistakes the user might choose to give his consent to be less often notified and leave a higher degree of autonomy / automation to the AI.
So the short answer to your question: It is a mixture of multiple patterns, not only one.
That is also how our conceptual model here needs to be understood. It represents a puzzle where different pieces are put together to create a holistic experience.
Thanks for the insight!