Skip to Content

Do you need to train people in how to use SAP software better? Or are you involved in other kinds of training projects within the SAP ecosystem? Whatever training you are doing, a key challenge is how to measure the effectiveness of your training intervention. How can you demonstrate that the money and time spent on training is achieving a useful effect for the business? 

To quote an IDC report that rates SAP as a market leader in IT training – “Overall, vendors are weakest at measuring the impact of training on the enterprise, making it difficult to effectively describe the opportunity cost of spending a dollar on training as opposed to some other essential need”

/wp-content/uploads/2012/03/kirkpatrick_82259.png

The answer to this 10 years ago would be to use something called the Kirkpatrick model (see diagram right) where you evaluate training in four levels: reaction, learning, behavior and results. His model has been hugely influential in the evaluation of workplace learning and the use of assessment.

But the Kirkpatrick model has flaws; in particular there is no logical connection between the levels. Just because trainees learn knowledge, it does not necessarily mean it produces results. A recent innovation is the A-model, a new way of thinking about assessment and evaluation of workplace learning. The A-model comes from Dr Bruce C. Aaron, whose career spans the Florida Department of Education and working in evaluation at Accenture as well as volunteering at the chapter and national level with ASTD.

The A-model does not start with the question: “How do you evaluate the effectiveness of training?”. Instead, it puts the training in context, by starting with the 3 P’s: Problem, Performance and Program.

The basic concept is that in order to make sense of evaluating something, you have to start with the business Problem that you are trying to solve. Problems need to be important for the business purpose and they must be measurable – for example a need to increase production, ship orders faster, reduce errors, increase revenue or heighten customer satisfaction. Deploying SAP software will often be to solve such a problem.

Once you have a clear definition of the Problem (and how you will measure if it is solved), you need to define the human Performance that will solve the Problem. Performance itself will typically be behavior that is directly tied to important accomplishment or results in the workplace. To get performance, you use Performance Enablers – new learning of knowledge, new skills, changed attitude, feedback or incentives, performance support or new equipment. For example, you might identify that people need to be more skilled with SAP software to solve the problem.

Only after defining the Problem and the Performance, should you define the Program. The Program consists of the intervention or solution to the problem, and might be a training course, some other kind of learning or improvement or performance support.

Just as in the picture below, the Program is just the tip of the iceberg, underpinned by the Performance and the Problem. Without them, evaluation of the training Program is not meaningful or useful. 

amodel iceberg.png

Thus if your program is a training course or learning intervention, you evaluate its effectiveness by measuring not only the delivery of program itself but also determining whether it achieves the human performance improvement you are looking for and has a positive impact on the problem it is there to solve.

The A-model starts with a business problem and helps you define how to improve performance to solve the problem, and then measure training or other performance support in that context.

I hope you find the A-model an interesting way of thinking about the effectiveness of training. I’ll write a follow-up post to explain how the A-model informs the processes of analysis, design and evaluation. In the meantime, you may want to listen to this 8-minute podcast from Dr Aaron or read his thought-provoking A-model white paper available here (free with registration).

To report this post you need to login first.

6 Comments

You must be Logged on to comment or reply to a post.

  1. Jenny Wang

    Thanks John for the sharing of A-model. In other words, if we want to ensure the success of this measuirement, it also requires users to be very clear about their learning objectives when then request a training, otherwise this methodology seems difficult to be achieved.

    (0) 
    1. John Kleeman Post author

      Jenny

      Thanks for the question.

      The A-model suggests that you measure effectiveness of training based on it solving a business problem, so yes users should be clear about their learning objectives, but also that these learning objectives are relevant to solving a problem that the business. See also the second part of this blog article at http://scn.sap.com/community/training-and-education/blog/2012/04/03/how-to-use-the-a-model-to-evaluate-a-sap-training-programme.

      John

      (0) 
      1. Jenny Wang

        Thanks John, I just nocited your 2nd blog post on A-model, and realized that A-model is more related to creating and developing a training’s perspective, not for a trainer to evaluate if this training has been successful or not~

        (0) 
        1. John Kleeman Post author

          Yes, it’s more related to a training manager or business leader perspective. But I think the ideas potentially useful to a trainer to think through how their training is related to business issues, even if it’s not directly applicable.

          (0) 
          1. John Kleeman Post author

            I mentioned this thread to Bruce Aaron of Ametrico (the inventor of the A-model) and he suggests:

            Anyone with a role in designing, delivering, or managing training should be able to frame up their individual function within this model.  Trainers and other Training Delivery Support roles naturally tend to be concerned about issues and information in the Program and Performance Enablement domains, but these folks should know where their issues and roles fit in to the big picture.  A trainer should be prepared to talk to a Training Director, or CLO, about how their training relates to Performance and the Business Problem.  We should understand those domains, even if we are unable to evaluate within them.  I.e., we should have a conceptual model for how all these pieces fit together, even if we make a decision to not test the model with data (e.g., to not evaluate ROI).  This is what the “alignment” benefit of the A-model implies – all roles and functions (planners, designers, delivery support, trainers, course sponsors, managers, training senior executives) have this common framework that links efforts across the domains back to the fundamental business issue.

            (0) 

Leave a Reply