Third in a series of blog posts on how an SAP LMS (LSO or SuccessFactors Learning) and SAP Assessment Management can help make trustable assessments.
In my first post in this series What makes an assessment result trustable?, I explained that to trust assessment results, assessments need to be reliable (consistent) and valid (measure what they intend to measure). And in my next post Top 5 tips for trustable assessments - Part 1, I shared five tips about how you could improve trustworthiness of assessments while authoring. Here are five further tips.
1. Use observational assessments to help measure performance
When assessing practical and communication skills, it’s often more valid to have someone observe the performance of the skill, as well as or instead of directly asking questions. For example, if you need to assess the ability for someone to weld a joint, perform a medical procedure or interview a customer, using an observational assessment lets you directly assess that practical skill.
SAP Assessment Management enables observational assessments, where a supervisor or instructor uses a tablet, smartphone or laptop, observes an employee performing a skill and rates him/her on performance. Such assessments like the one in the graphic often use checklists or objective scales of performance and can be very effective in measuring workplace performance.
2. Pilot assessments well
Before you can trust an assessment, you need to pilot it and review the results of the pilot prior to using it for real. There are a variety of methods of piloting, including “beta” programs, getting instructors or other experts to take the assessments and allowing test takers to comment on beta questions.
One useful and easy mechanism for ongoing assessment programs is to include new, experimental questions within production assessments – these don’t count towards the score but give you data from real test-takers to check the questions work well before you finalize them.
3. Deliver assessments robustly
Once an assessment has been created and piloted, you need to ensure the assessment delivery process has integrity so that validity and reliability are maintained.
If you are using SAP LSO or SuccessFactors Learning, you can easily call SAP Assessment Management from courses or other learning, robustly start the assessment and securely pass the result back in the LMS. There are a host of detailed capabilities within both LMS and assessment management system to help make this effective including consumer quality user interfaces and robust means to deal with computer or network failures but still record results.
4. A lockdown browser can promote exam integrity and security
There are many ways to improve security. One helpful tool is a “lockdown browser”. This makes it harder for a participant to browse the internet or use chat sessions to seek advice from others whilst taking an assessment. It also protects against printing the questions or otherwise passing them to others.
If you use SAP LSO or SuccessFactors Learning to call tests or exams using SAP Assessment Management, it’s easily possible to arrange for a lockdown browser “Questionmark Secure” to be used for the assessment. The employee uses a normal browser for learning, and is transferred seamlessly to the lockdown browser to take the test or exam.
5. Use item analysis reports to weed out poor questions
My last tip is to run item analysis reports regularly.
A poorly worded or ambiguous question or one that isn’t aligned with your assessment objectives can significantly reduce the validity and reliability of the assessment. Item analysis reports take just moments to run and can flag questions for review as potentially poor questions. For example in the example report below, 4 questions are tagged as amber meaning that they need review and might need removing from the assessment or changing.
If you regularly run item analysis reports and remove or improve poor questions, your assessments will become more trustable – and if you don’t do this, you need to be careful about relying on assessment results.
I hope the 5 tips above and from my previous post will help you create and deliver more trustable assessments.
I hope also that if you are using SAP LSO or SuccessFactors Learning, you will see that adding SAP Assessment Management will make it easier to trust your assessments results and rely on the decisions you make based on them. You can see more about SAP Assessment Management on SAP's website.
- SAP Managed Tags:
