First in a series of blog posts on how an SAP LMS (LSO or SuccessFactors Learning) and SAP Assessment Management can help make trustable assessments.
How can you trust an assessment result to make business decisions on whether an employee is competent? If you use quizzes, tests or exams to check employee knowledge or skills, what makes them trustworthy?
I went to the doctor a few days ago with a minor ailment. The doctor wanted to rule out something and arranged a blood test. The blood test came up negative, and he and I completely believe in and trust the results of that test. For a small amount of hassle and work, my doctor can diagnose and treat me effectively.
The ideal with workplace assessments is the same. We want to be able to rely on assessments to identify knowledge, skills and abilities in a completely trustable manner.
If assessment results are trustable, you can:
- Reduce regulatory compliance risk and fines by identifying poor practices and lack of knowledge before it impacts your business
- Reduce risks to personnel by testing on health and safety procedures
- Get the business benefits of competence in customer service, manufacturing quality and other aspects of your business that rely on people
- Rely on people to use SAP or other software effectively
- Identify training needs to use resources wisely to train effectively
- Make better decisions in talent management: recruiting, promoting and developing talent
- Onboard new employees well
- Set up partner programs to verify knowledge and skills of your sales and technical channels
- Develop certification programs for your employees, customers or partners that add value to your product offering
- Use “big data” technology to correlate assessment results with performance, which is only possible if you can trust your assessment results
As a corollary, if your assessment results are not trustable, you run the risk you will:
- Hire the wrong people
- Promote the wrong people
- Run a legal risk from making employment decisions that are not defensible
- See errors in health and safety, manufacturing, customer service and regulatory compliance that lead to reputation loss and/or compliance fines
- Lack evidence employees are competent to perform tasks safely and in compliance with regulations
- Waste time in training people what they already know
- Fail to train people in what they need to know
- Have people unable to use or gain value from SAP or other software
- Devalue any partner or certification programs that use assessments
- Draw incorrect conclusions from assessment results: “garbage in means garbage out”
- Waste the time spent in creating assessments and delivering them
How can you trust assessment results? The two keys are reliability and validity.
Reliability explained
An assessment is reliable if it measures the same thing consistently and reproducibly. If you were to deliver an assessment with high reliability to the same participant on two occasions, you would be very likely to reach the same conclusions about the participant’s knowledge or skills. A test with poor reliability might result in very different scores across the two instances.
An unreliable assessment does not measure anything consistently and cannot be used for any trustable measure of competency. It is useful visually to think of a dartboard; in the diagram to the right, darts have landed all over the board—they are not reliably in any one place.
In order for an assessment to be reliable, there needs to be a predictable authoring process, effective beta testing of questions, trustworthy delivery to all the devices used to give the assessment, good-quality post-assessment reporting and effective analytics.
Validity explained
Being reliable is not good enough on its own. The darts in the dartboard in the figure to the right are all consistently in the same place, but not in the right place. A test can be reliable but not measure what it is meant to measure. For example, you could have a reliable assessment that tested for skill in word processing, but this would not be valid if used to test machine operators, as writing is not one of the key tasks in their jobs.
An assessment is valid if it measures what it is supposed to measure. So if you are measuring competence in a job role, a valid assessment must align with the knowledge, skills and abilities required to perform the tasks expected of a job role. In order to show that an assessment is valid, there must be some formal analysis of the tasks in a job role and the assessment must be structured to match those tasks. A common method of performing such analysis is a job task analysis, which surveys subject matter experts or people in the job role to identify the importance of different tasks.
Trustable assessments must be reliable and valid
Trustable assessments must be reliable AND valid. The darts in the figure to the right are in the same place and at the right place. When you are constructing an assessment for competence, you are looking for it to consistently measure the competence required for the job.
Unreliable, invalid assessments can help a bit in learning. Giving people questions as retrieval practice helps prevent forgetting. And if people know that there is a test or quiz after learning, it can encourage them to study for the test or quiz even if the test is poorly designed. However, for any kind of decision-making based on assessments, you need to be able to trust the results, which means that you need reliable and valid assessments.
Within the world of health care, a great deal of effort goes into making sure that blood tests are both reliable (consistent) and valid (measure what they are supposed to measure). And the benefits are immense.
In the same way, if you can make your assessments reliable and valid, then the benefits are huge. In my next post in this series, I’ll explain some of the key things you need to do to make assessments reliable and valid and how using an SAP Learning Management System together with SAP Assessment Management by Questionmark can get you there.
- SAP Managed Tags:
1 Comment
