Remote usability testing piloted at this year’s virtual SAPPHIRE NOW and ASUG Annual Conference. From June 29 until July 1, 2020, 14 product teams tested their applications with more than 130 participants – with great impact: Besides plenty of positive feedback, more than 60 backlog items were collected to ensure product excellence. Read about the experiences of the first virtual usability testing program, its challenges, and lessons learned.
Usability testing helps to understand the users’ needs and the way they work. Working closely together through each phase of development ensures that the most usable products and solutions with the best user experience end up in the hands of our customers.
Usually, usability testing takes place on-site at events. Since there are currently no in-person events for 2020 due to Covid-19, usability testing took place remotely, requiring a completely different set-up. “We did not only test apps, but also the remote set-up as an important step towards the future of usability testing”, summarizes Michelle Clary, Marketing Expert in SAP User Experience.
Image Source: SAP Image Library. Graphics: Created by SAP User Experience
One clear benefit of going virtual was that testing participation was not only open to SAPPHIRE attendees, but to all SAP customers and partners around the world. With more than 320 registrations, participants came from various backgrounds, like development, consulting, and sales while representing a wide range of user profiles (managers, project leads, IT decision makers, etc.). During one-on-one 60-minute sessions, they could try out new and existing software and gave feedback directly to the product teams that build them.
Image Source: SAP Image Library. Graphic: Created by SAP User Experience
The post-test questionnaire showed that the flexibility in terms of time and the independence of location were greatly appreciated: “Overall, the remote sessions are great. They are wonderful, they are easy. I couldn’t see them happening any other way.” (SeaPort Sound Terminal LLC).
Besides the questionnaire, the turnkey remote usability program included the creation of a registration platform and a step by step testing protocol via Zoom incorporating non-disclosure agreements (NDAs), General Data Protection Regulation (GDPR) and compliance regulations. Also, support in preparing the sessions was part of the ‘worry-free’ package.
For the sessions, at least one moderator, one notetaker and one other team member to monitor the new Event Scheduler Tool were needed. Performing dry runs close to the day of the session allowed to test the technical infrastructure, to validate that all the tasks could be performed and to eliminate any possible issues. For example, sending tasks via chat did not work as expected since they got in the way of screen sharing.
Preparation also included an introduction to Zoom which has proven itself as a good conferencing tool for conducting virtual testing as it allows participants to change their profile to an anonymous username to protect their identity. With remote testing comes the great option of voice and video recording, yet GDPR compliance is a must: Names are not allowed to be on record and recordings must be deleted after 90 days. However, most teams decided to turn on the camera only for the personal introduction. During testing, it made sense to turn it off in favor of a more stable connection and less distraction for the participants.
How far away from your fridge would you pin your shopping list? Metaphorically speaking, that’s what one of the topics was about – the evaluation of new personalization options for SAP Fiori 3 tables and the filter bar. Due to a similar target group, there was a combined test in which the two concepts were tested with two prototypes. The sessions took place via Zoom and screen sharing with fully interactive Axure RP Pro / Team prototypes. For this, Rainer Filsinger, Björn Bader, and Lisa Jaeger alternated their roles as moderators and note-takers for each test.
“Despite the remote setup, the tests worked well, and we were able to gather valuable feedback”, reflect the designers. “Participants could follow our concepts and mostly appreciated the new personalization paradigms.” In particular, they highly valued the table personalization settings being in one popover in the column headers and the second entry point via the settings icon. Also, the table’s context-sensitive quick actions (filter, sort…) were considered as very useful.
A popover directly opens in the column header (1) to sort data. The settings icon (2) provides the same functionality.
Testing the filter personalization, participants explicitly complimented the message toast after switching the filtering mode: „Very good is the message toast as a system reaction. Excellent!“ Yet, the refresh icon was a bit confusing.
A message toast (3) informs that the selected filtering mode will be applied automatically. The refresh icon (4) for the mode switcher wasn’t intuitive enough.
Uncovering negative findings is as important as positive findings. For example, sort indicators and options for filter drag and drop were missing. For some participants, the refresh icon for the mode switcher was not intuitive enough.
And what are Björn’s tips for their next remote usability test? “We’ll see how testing works when the camera is turned on as the users’ body language helps us to understand their behavior. Plus, we would like to try out click animations to monitor clicks better.”
Incorporating usability testing into your development process ensures that SAP continues to deliver the highest caliber of products and solutions. However, when working with customers, it is very important you keep SAP and its customers protected by using tools and services that are GDPR compliant.
Do you have any questions about usability testing at SAP? Simply follow me in the SAP Community.
For information about the bigger picture of the role of user research in the product development life-cycle, watch the following video explaining how user research helps to foster customer success:
“Please always enable remote access”
Usability testing helps to understand the users’ needs and the way they work. Working closely together through each phase of development ensures that the most usable products and solutions with the best user experience end up in the hands of our customers.
Usually, usability testing takes place on-site at events. Since there are currently no in-person events for 2020 due to Covid-19, usability testing took place remotely, requiring a completely different set-up. “We did not only test apps, but also the remote set-up as an important step towards the future of usability testing”, summarizes Michelle Clary, Marketing Expert in SAP User Experience.
Image Source: SAP Image Library. Graphics: Created by SAP User Experience
One clear benefit of going virtual was that testing participation was not only open to SAPPHIRE attendees, but to all SAP customers and partners around the world. With more than 320 registrations, participants came from various backgrounds, like development, consulting, and sales while representing a wide range of user profiles (managers, project leads, IT decision makers, etc.). During one-on-one 60-minute sessions, they could try out new and existing software and gave feedback directly to the product teams that build them.
Image Source: SAP Image Library. Graphic: Created by SAP User Experience
The post-test questionnaire showed that the flexibility in terms of time and the independence of location were greatly appreciated: “Overall, the remote sessions are great. They are wonderful, they are easy. I couldn’t see them happening any other way.” (SeaPort Sound Terminal LLC).
“Tremendous job in getting all of us ready”
Besides the questionnaire, the turnkey remote usability program included the creation of a registration platform and a step by step testing protocol via Zoom incorporating non-disclosure agreements (NDAs), General Data Protection Regulation (GDPR) and compliance regulations. Also, support in preparing the sessions was part of the ‘worry-free’ package.
For the sessions, at least one moderator, one notetaker and one other team member to monitor the new Event Scheduler Tool were needed. Performing dry runs close to the day of the session allowed to test the technical infrastructure, to validate that all the tasks could be performed and to eliminate any possible issues. For example, sending tasks via chat did not work as expected since they got in the way of screen sharing.
Preparation also included an introduction to Zoom which has proven itself as a good conferencing tool for conducting virtual testing as it allows participants to change their profile to an anonymous username to protect their identity. With remote testing comes the great option of voice and video recording, yet GDPR compliance is a must: Names are not allowed to be on record and recordings must be deleted after 90 days. However, most teams decided to turn on the camera only for the personal introduction. During testing, it made sense to turn it off in favor of a more stable connection and less distraction for the participants.
Exploring new personalization patterns for SAP Fiori 3
How far away from your fridge would you pin your shopping list? Metaphorically speaking, that’s what one of the topics was about – the evaluation of new personalization options for SAP Fiori 3 tables and the filter bar. Due to a similar target group, there was a combined test in which the two concepts were tested with two prototypes. The sessions took place via Zoom and screen sharing with fully interactive Axure RP Pro / Team prototypes. For this, Rainer Filsinger, Björn Bader, and Lisa Jaeger alternated their roles as moderators and note-takers for each test.
“Despite the remote setup, the tests worked well, and we were able to gather valuable feedback”, reflect the designers. “Participants could follow our concepts and mostly appreciated the new personalization paradigms.” In particular, they highly valued the table personalization settings being in one popover in the column headers and the second entry point via the settings icon. Also, the table’s context-sensitive quick actions (filter, sort…) were considered as very useful.
A popover directly opens in the column header (1) to sort data. The settings icon (2) provides the same functionality.
Testing the filter personalization, participants explicitly complimented the message toast after switching the filtering mode: „Very good is the message toast as a system reaction. Excellent!“ Yet, the refresh icon was a bit confusing.
A message toast (3) informs that the selected filtering mode will be applied automatically. The refresh icon (4) for the mode switcher wasn’t intuitive enough.
Uncovering negative findings is as important as positive findings. For example, sort indicators and options for filter drag and drop were missing. For some participants, the refresh icon for the mode switcher was not intuitive enough.
And what are Björn’s tips for their next remote usability test? “We’ll see how testing works when the camera is turned on as the users’ body language helps us to understand their behavior. Plus, we would like to try out click animations to monitor clicks better.”
Evaluate the usability of your product
Incorporating usability testing into your development process ensures that SAP continues to deliver the highest caliber of products and solutions. However, when working with customers, it is very important you keep SAP and its customers protected by using tools and services that are GDPR compliant.
Do you have any questions about usability testing at SAP? Simply follow me in the SAP Community.
For information about the bigger picture of the role of user research in the product development life-cycle, watch the following video explaining how user research helps to foster customer success:
- SAP Managed Tags:
