Putting the New Quiz Builder to the Test in SuccessFactors Learning
A lot of visibility has been placed on the new People Profile from SuccessFactors in the core SF platform/EC product with the b1508 release, and rightly so. The new profile based on the latest SAP Fiori/UI5 design principles provides a more modern and efficient user-friendly interface when displaying employee profile details. Not to be overshadowed by the new People Profile though is the new Quiz Builder functionality that is now part of SuccessFactors Learning. The Quiz Builder is meant to replace (eventually) the legacy Plateau Question Editor (PQE) used in the LMS to build questions and exams. For all the LMS admins out there, this could be one of the best updates in the LMS in a while, and there have been some rather nice features introduced as of late in their own right. The PQE product though is an eye sore and a chore to deal with at times. While the new People Profile in the SuccessFactors core platform might be the appointed star of the show, the bigger leap in terms of UX design improvement and usability is easily won by the Quiz Builder in the LMS (if this were a competition), at least in my opinion. In this blog I explore what the new Quiz Builder brings to the LMS and what it means for the future of assessments in SuccessFactors Learning.
Quiz Builder vs. PQE – Why the Fuss?
I attended SuccessConnect 2015 in Singapore just before writing this blog and set in on a session given by the Director of Learning Products and Strategy at SuccessFactors for the APAC region. In discussing the LMS roadmap and latest features in the system, you could tell that she was easily the most excited about what the new Quiz Builder brings to the LMS. She also made a very interesting comment, and hopefully I paraphrase her correctly here. She stated that of all the usability testing that SuccessFactors has done on their products, the new Quiz Builder in SuccessFactors Learning has scored the highest among their testing groups. Hopefully that alone peaks your interest to continue reading?
First things first, a quick overview of what the Quiz Builder is in the LMS. With this new tool, LMS administrators now have an alternative to creating and building quizzes to deliver to learners. Previously this was done exclusively through the aforementioned PQE, where admins could create Questions and Exams for the same purpose. Let’s get clear on the semantics here now as this appears to be how SuccessFactors is going to refer to these going forward. ‘Exam’ is the old term for assessments built through the PQE while ‘Quiz’ is the new term for assessments built through this new tool, termed the Quiz Builder. This initial release of the Quiz Builder will allow for the creation of quizzes for the most commonly used question types (multi-answer, single-answer, and true/false). Basic settings for the quiz as a whole can be specified as well, including passing percentage and quiz attempts.
If you picked up on a previous line in my opening paragraph here, I noted the new Quiz Builder is meant to replace the PQE at some point. As with a lot of features SF introduces, a base level of functionality is rolled out initially and then you will see the subsequent releases bringing increased features related to the update in order to enhance the capabilities further (in the case of the LMS, see the recently introduced and progressively improved Programs and Open Content Network). With that in mind, the following points below should be considered.
- The Quiz Builder will serve as the basis for the replacement of the legacy Plateau Question Editor (PQE).
- Both the PQE and the Quiz Builder will be available for use for the foreseeable future. PQE will not be retired any time soon, especially not before the Quiz Builder provides all the features and capabilities that the PQE currently entails (more on the current gaps later).
- Subsequent releases after b1508 are planned to focus on supporting new question types and further usability enhancements.
Creating Quizzes in the New Quiz Builder
The new Quiz Builder can be accessed now under the LMS admin portal under the Content -> Quizzes section. For those familiar with creating Programs in the LMS, creating Quizzes through the Quiz Builder will look much the same. The quiz as a whole is summarized in one screen, providing a more complete overview of the quiz and questions utilized than ever was possible in the PQE. The PQE itself is used to create questions for use in Exams, however, admins had to perform these two tasks separately (create questions vs. creating exams). The new Quiz Builder brings the two concepts together, as they should be, and not requiring the two to be maintained outside of the central concept of the quiz. The matrix below provides further details on the possible settings to be made for a quiz while the screen shot gives you an idea of what the UX looks like for managing a quiz.
Quiz Administration Overview
Quiz Settings
Section |
Settings |
General |
Quiz ID (unique ID for quiz) Quiz Name Description |
Questions |
Total number of questions in quiz including numbers of questions by type (True/False, Single Answer, Multi Answer) |
Messages |
Welcome Message (upon start of exam to learner) Success Message (upon successful completion of quiz) Failure Message (upon failure of quiz) |
Settings |
Quiz Type (Sequential or Free-Form) Passing Percentage Feedback Display (No feedback, feedback only, or feedback+correct answer choices) Active (yes/no) Results summary provide (yes/no) Enable quiz review (yes/no) Quiz Attempts Limit |
Locales |
Supported locales (translations) for quiz and to which locales the quiz is currently published. |
Of note from above, the capability to build quizzes and provide locale based text translations for quiz title/description, messages, and question/feedback is a HUGE new feature previously not possible in the PQE. For clients who need to build quizzes in multiple languages, managing this in one quiz as opposed to numerous questions per language requiring separate exams per language is a much needed improvement from days past.
Building questions into the quiz is also much improved and more efficient in the Quiz Builder when compared to the PQE. As previously noted within the PQE, questions had to be created independently (in a painful and inefficient editor to boot), then questions had to be pooled into Exam objects as well. Within the Quiz Builder, questions can be created as part of the overall quiz directly from the Question maintenance screen and ordered accordingly. With the new SAP Fiori/UI5 UX, the process of creating new questions, adding/adjusting question text and answer choices, and ordering the overall questions in the quiz is so much more smooth than the PQE that the LMS admins of old (back in the Plateau days) may just shed a tear or two during your first go with the application.
Question Management Within a Quiz
The experience only gets better when learners take the actual exams within the LMS as well. While the legacy exams would get the job done, they weren’t the prettiest things to look at or the most modern. The new Fiori UX principles and simplicity of the design does make the new quizzes a pleasure to run. They are sleek and minimalist, keeping the focus on the assessment where it should be but at the same time not making the user feel like they have jumped into an early 2000’s web application in order to assess their understanding of the subject matter in question.
Examples of Questions Within Quiz (Learner)
Quiz Completion / Results (Learner)
Mind the Gap – Where the New Quiz Builder Lacks vs. PQE
As with most new features from SuccessFactors across the various modules, the new Quiz Builder is not meant to be a full replacement to the existing PQE across all capabilities, at least not yet. There are some important gaps to consider presently with the b1508 release when comparing like-for-like against the PQE. The list below is not comprehensive, but points out some of the key gaps to be aware of. Keep an eye on the next few releases for SuccessFactors Learning as I expect many of these gaps to be eliminated as the tool matures.
- Adaptive learning and use of objectives
- Questions in the new Quiz Builder are built directly within a quiz itself and cannot be assigned to Objectives within the LMS.
- Reuse of questions across quizzes
- The PQE is focused on building a question bank so to speak, then assigning questions to objectives within the LMS and leveraging them where desired in Exams. This can allow admins to use questions/objectives across multiple exams. Currently in the new Quiz Builder, questions are created directly within a quiz and unique to that quiz only and cannot be automatically reused as is in another quiz.
- Randomizing questions within quiz
- While you can randomize the answer choices within questions in the new Quiz Builder, much like the PQE, you won’t be able to randomize the order of the questions within the quiz itself.
- Question variants and revisions
- The PQE allows you to create different variants of a particular question that can then be randomized for which variant is displayed to a user when they take an exam. The Quiz Builder does not currently provide the capability to create different variants of a single question for use within a quiz.
- Question revisions are also not supported currently in the new Quiz Builder. Within PQE you can update questions and save revisions of questions which comes in handy when an exam is active and being used. Old question revisions can be maintained while new revisions can be used in future exams. I haven’t played around with the Quiz Builder to the extent yet to see how/if it would allow for question revisions with respect to publishing quizzes in certain locales, so something to watch out for.
- Images in questions
- PQE allows you to upload images to use for questions. Currently the Quiz Builder does not support images or other media for use while presenting a question to a user.
The Future of Quizzes in SuccessFactors Learning
While yes there are functionality gaps with the new Quizzes in respect to the legacy PQE tool in SuccessFactors Learning, the UX and reduction in administration efforts for managing quizzes is a huge step forward for LMS admins. The capabilities of the tool currently are enough to create general quizzes for a range of purposes, however you may need to wait on the next few releases if you wish to create more rich assessment experiences with the new quiz functionality.
My hope is SuccessFactors does not stop with the new Quiz Builder in the LMS once it reaches parity with the legacy PQE. It would be great to see additional question types being supported which were not previously (matching for instance). It’s not uncommon to see clients stick to building assessments in 3rd party tools to create more robust assessment experiences overall. If SuccessFactors could bridge the gap between the provided LMS Quiz functionality and those best of breed e-learning content authoring tools with respect to assessment functionality, that would be a great advancement for the LMS in the future. Regardless, it is great to see new and improved LMS capabilities from SuccessFactors Learning in the new Fiori UX design, which I hope is only a precursor of things to come.
One last note, the Quiz Builder is not the only Fiori designed UX tool within the LMS to make a debut with the b1508 release. SuccessFactors has also introduced a new Instructor View based on this design as well, which looks to be a great new tool for long overdue functionality for instructors that has been missing in the LMS. A topic for another day perhaps, but not to be overlooked…
What are your thoughts on the new Quizzes in SuccessFactors Learning?
Great Post Eric. It is also worth noting that once the quiz has been created (and domain selected) there is no way to update this with a different value.
Very good point Travis and one that I had glossed over. Domain isn't part of the General Quiz settings so I guess you better pick the right one when you create a quiz initially. 🙂 Hopefully that is added in there as something that can be modified in a later release.
Great blog Eric...Indeed!!!
I have two points to be mentioned here.
1. I worked on both Exam/Quiz tool and what I noticed is that it does not currently accommodate open ended answers functionality. Survey does , but the survey is not triggered until after the learning event is completed.
I have my finger crossed that the new quiz tool just released will eventually support open ended answers too..
2. Migration of question created PQE: All the question that we created with PQE(former tool) be automatically transferred/Created to New Quiz Tool.
Hoping this all will be there eventually in QUIZ functionality... 🙂
Many Thanks!!
Regards,
Ritesh Singhvi
Ritesh
Open ended answers would be a bit tricky. Quizzes, like Exams, are currently setup to be auto-graded, which is the main reason why we don't see open ended questions as trying to have them 'systematically' evaluated would be a challenge. Surveys yes capture the open ended feedback where desired, but their purpose is different.
Now maybe in the future SF expands the ability to have Quizzes be auto-graded or manually graded by an admin/instructor, and then if being manually graded you're allowed to have open ended questions in the quiz.
I agree with your migration point for legacy PQE questions, and maybe that will come. I'm sure there are plenty of technical challenges with that though, so not sure I'd bank on it, but you never know.
Eric
Hi Eric,
Thanks for sharing the updates of the new & improved quiz builder option in SF LMS. Specially the discussion around the comparison with the existing PQE and Quiz builder. It is good that you have shared some light on what are the current gap in 1508 test release and what would be the future ahead for SF Quiz functionality.
And yes, the points mentioned by Ritesh on open ended questions and migrating existing PQE exams to Quiz builder would definitely be an advantage in future. 🙂
Regards,
Nithin
Eric -- thanks for the great write up!
The interface is very easy to us. Although just one hint for set up - be sure to use the Locales > Publish option to make your Quiz available for users.
One other capability that PQE supports is use of a spreadsheet to import questions. That is a great feature - but probably only feasible as long as the questions are independent of the exam.
And PQE does support a robust level of analysis at the question level. I am not sure that most organizations use that feature, but it can be useful for validation.
Cheers,
Lane
SF LMS Instructor/Instructional Designer (now retired)
Hi Eric,
Great writeup. Another thing to add to the quiz wishlist here would be question pooling and duration.
Regards,
Anand
Great info. Do you know if a quiz built in Quiz Builder allows for tracking/reporting on individual learner responses?
Hi there. I just created a Quiz in Quiz Builder and have set it to allow unlimited attempts. When I test it however, I don't get a Retry Quiz option when I fail. I have to Exit the Quiz and then relaunch to attempt again. Is it supposed to function this way?