The A-model, a new way of thinking about measuring the effectiveness of training

Do you need to train people in how to use SAP software better? Or are you involved in other kinds of training projects within the SAP ecosystem? Whatever training you are doing, a key challenge is how to measure the effectiveness of your training intervention. How can you demonstrate that the money and time spent on training is achieving a useful effect for the business? 

To quote an IDC report that rates SAP as a market leader in IT training – “Overall, vendors are weakest at measuring the impact of training on the enterprise, making it difficult to effectively describe the opportunity cost of spending a dollar on training as opposed to some other essential need”

The answer to this 10 years ago would be to use something called the Kirkpatrick model (see diagram right) where you evaluate training in four levels: reaction, learning, behavior and results. His model has been hugely influential in the evaluation of workplace learning and the use of assessment.

But the Kirkpatrick model has flaws; in particular there is no logical connection between the levels. Just because trainees learn knowledge, it does not necessarily mean it produces results. A recent innovation is the A-model, a new way of thinking about assessment and evaluation of workplace learning. The A-model comes from Dr Bruce C. Aaron, whose career spans the Florida Department of Education and working in evaluation at Accenture as well as volunteering at the chapter and national level with ASTD.

The A-model does not start with the question: “How do you evaluate the effectiveness of training?”. Instead, it puts the training in context, by starting with the 3 P’s: Problem, Performance and Program.

The basic concept is that in order to make sense of evaluating something, you have to start with the business Problem that you are trying to solve. Problems need to be important for the business purpose and they must be measurable – for example a need to increase production, ship orders faster, reduce errors, increase revenue or heighten customer satisfaction. Deploying SAP software will often be to solve such a problem.

Once you have a clear definition of the Problem (and how you will measure if it is solved), you need to define the human Performance that will solve the Problem. Performance itself will typically be behavior that is directly tied to important accomplishment or results in the workplace. To get performance, you use Performance Enablers – new learning of knowledge, new skills, changed attitude, feedback or incentives, performance support or new equipment. For example, you might identify that people need to be more skilled with SAP software to solve the problem.

Only after defining the Problem and the Performance, should you define the Program. The Program consists of the intervention or solution to the problem, and might be a training course, some other kind of learning or improvement or performance support.

Just as in the picture below, the Program is just the tip of the iceberg, underpinned by the Performance and the Problem. Without them, evaluation of the training Program is not meaningful or useful. 

Thus if your program is a training course or learning intervention, you evaluate its effectiveness by measuring not only the delivery of program itself but also determining whether it achieves the human performance improvement you are looking for and has a positive impact on the problem it is there to solve.

The A-model starts with a business problem and helps you define how to improve performance to solve the problem, and then measure training or other performance support in that context.

I hope you find the A-model an interesting way of thinking about the effectiveness of training. I’ll write a follow-up post to explain how the A-model informs the processes of analysis, design and evaluation. In the meantime, you may want to listen to this 8-minute podcast from Dr Aaron or read his thought-provoking A-model white paper available here (free with registration).