Thought leader interview – SAP Certification Manager Sue Martin on Transforming the SAP Certification Programme

Interview by John Kleeman, Questionmark Chairman and assessment expert, with Sue Martin from SAP, on how she and colleagues are making the SAP Certification Programme a key enabler for successful SAP projects. Why certification matters to SAP, SAP customers and SAP specialists.

John : Sue, what is your job role at SAP?

Sue : I am Global Certification Manager, so I’m responsible for the Certification Programme and Strategy at SAP worldwide. I work together with other teams within SAP to ensure that we get the exams out, we’ve got the right strategy in place and that the certifications focus on the right priorities.

John : How did you come to join SAP?

Sue : I originally came to SAP four years ago as a freelance consultant. SAP’s certification programme had been around since the mid-90s, but it hadn’t changed a lot in the previous 10 years. I was asked to review and identify ways we could enhance the programme.

The market for SAP had changed immensely, and the certification programme hadn’t really changed to match the market needs. So I did an analysis and review, spoke with a lot of stakeholders and customers, and made 10 recommendations of how we should change the certification programme. In a nutshell the change was moving from certification as a post training assessment, which validated that you knew what you’d learned in classroom courses, towards something that provides value to the individual and a benchmark on which customers can base important decisions.

After I’d done my review, the majority of the recommendations were accepted, and that was when I was asked to come aboard to SAP permanently to implement it. And that’s what we’ve been doing for the last 3 years.

John : What is SAP’s motivation, is to make money from certification as a revenue stream or to make SAP projects and customers more successful?

Sue : Only the second!

The post training assessment side of things was proving the return on investment in education, but we needed to take a step back and look at the big picture. Certification should not be there just to comply with educational objectives, but it should have a higher mission, to drive quality into the market, to protect our customers and ensure their success with SAP software. That has been the basis of the transformation.

John : If you are an SAP customer, what is the benefit of recruiting people who have a certification?

Sue : One of the points of the transformation was that we took it beyond just the new hire certification. The old certification exams were very much aimed right at the beginning of your SAP career.

But we’ve changed the exams to be Job Task Analysis based, where certifications test what you need to know in the job.  It’s not just about knowing functions and features of SAP software, but it’s about what can you actually do, because this gives the customer more value.

As a customer, I’m not interested in bringing someone on board because he or she knows everything you ever wanted to know about SAP software plus lots that you don’t really need. I’m more interested as a customer in knowing exactly what that individual can do, what tasks can they perform, how can they help me ensure business success.

We also aligned the job roles being certified with needs, and validated this with partners and customers. We’ve made certifications more scenario-based, gone away a little from “which button do I press” to “how would I address that particular customer concern” or “how do I solve a particular customer issue”. And that adds immense value to me as a customer, to know that someone understands my business needs and can interpret those into software requirements.

The second step that enhanced the value for customers is that we’ve added the second, Professional layer to certifications. These are aimed at validating people with 4+ years of experience, who have got real business experience. We’ve now got Professional certifications in more or less every area. These are very much scenario based questions, all the questions are written by higher level consultants, senior consultants who are in the job, doing the job, faced with those business challenges every day. It’s not just knowledge recall or something you could learn out of a book. Bringing that level in, we’ve now had our first couple of thousand people running through the Professional certification programme, and I think that makes a huge difference.

John : What effort goes in at SAP to make the questions good quality and set the scoring fairly?

Sue : We follow industry standard practices to get legally defensible exams into place.  In order to get more experiential subject matter expertise, we moved from instructors writing questions to much more involvement from our own consulting organization. At the same time, to get the level of consistency and quality of review, we significantly increased the number of subject matter experts (SMEs) involved.

Here is an overview of the steps we follow:

1. The first step is job task analysis to identify the tasks that we ask questions on. These are validated with customer organizations, partners and with our own consulting organization.
2. The next step is the design phase, where we have the item writing workshops that write and review the questions. It depends on the subject matter, but there are normally at least 6 or 7 SMEs that sit in these workshops, supported by a psychometrician.
3. We then have peer review from another set of SMEs after the workshop. By SMEs I mean primarily people who are actually working in that area, so for a Professional certification that would be a Senior consultant or a Platinum consultant.
4. Then another group of SMEs go through the exam as beta testers and we look at the results.
5. Then a group of experts goes through the questions and do a pre-standard setting process, estimating the difficulty level of each question.
6. Then we have the final standard setting meeting to get to the cut score (pass score) determination phase.  And then at the end of all that, we finally have the exam!

John : I know you use Questionmark software in authoring and delivery of exams. Is our Questionmark technology working well for you?

Sue : One of the important parts of the transformation phase 1 was to be more flexible from a delivery perspective and with authoring tools. So we decided in 2007 to move over to an industry standard system (industry standard being the key to everything that we are trying to do in this project) rather than our previous in-house developed test delivery and creation tool. That was when we migrated over to Questionmark software.

We rolled out Questionmark Perception to over 80 SAP education centres worldwide and that now provides us with excellent content authoring tools which really enable us to author effectively, working with a large number of subject matter experts. And it allows us to have a great delivery tool for all of our SAP education centres. We also have in place test delivery via Pearson Vue centres around the world to increase accessibility throughout our Ecosystem.

John : Can you share how many people a year get certified? How many Professional and Associate?

Sue : We’ve got about 250,000 certified individuals around the world. We’re currently running at about 60,000 people a year getting certified.

The Professionals are growing hugely. We’ve got the first 2,000 Professionals, and a lot of that has been in the last 6 months to a year. So it’s quite a steep curve.

John : Do SAP employees themselves get certified?

Sue : Some time ago, experienced people used to say that they didn’t need certification, as they already had experience. When we transformed the certification, we said this was unacceptable – how could we as SAP go to the outside world without being certified ourselves, and say this is a credible programme?

So we sat down with our consulting organization and took in their feedback, and ensured we were working closely with them to ensure that our programme was really addressing their needs. And that in itself has been really good as there have been quite a few emerging roles that we’ve put into our certification programme that have come from our consulting organization. The business of technology evolves, so the collaboration with our consulting organization has been a win-win, it’s also meant that we’ve got 95% of our consulting organization certified in the last couple of years.

John : Do you have to take a specific training course before taking a certification, do some certifications require SAP or SAP partner training?

Sue : From a global perspective, no we don’t, there is no pre-requisite to take a training course first.  On-the-job experience can be equivalent to training.

We do have security issues in certain geographies, and in two countries in Asia, we do require training prior to certification as a security check to prevent inappropriate test takers. 

John :  You’ve been talking about transformation phase 1, what is phase 2?

Sue : Our aim for phase 2 is to make sure we are lifting the programme up to a different level, and that it’s really going to improve the quality of customer implementations. One of the things we are bringing into place in February is a Certification Influence Council (CIC) to help us make phase 2 of the transformation happen. This has been instigated by Bill McDermott our co-CEO and will have internal stakeholders from all parts of SAP and external stakeholders.

As well as internal stakeholders, the CIC will have representatives from user groups, partners, customers – and also two of the “certification 5”, industry analysts who’ve blogged about certification.  They will be meeting every quarter to review our direction and ensure that certification is really improving business success as a customer.

John :  What improvements are likely in phase 2?

Sue : Some of the things we are looking at: we want to create and nurture a certified community, so we need to have more benefits in place really for the individual, we are considering more flexible virtual learning opportunities for the individual and also more benefits for the certified community – access to events, access to peer exchange, more privileges in terms of discounts off our press books.

Then we’re looking at ways to improve the certification. One area to look at is live application testing, and we’ll be working with the CIC to see whether there are workable and scalable models we can use. We’re also looking at the Master level, which will be aimed at the crème de la crème, the real experts, who have 8+ years of experience. Now that the Professional level has got some l recognition in the market, we’ll be looking more closely whether or when we’ll introduce the Master level.

Then we will look at other exam development techniques, and I think Questionmark can help us with that, things like essay marking and more innovative item types. Also one of the things on our agenda is ways to provide testing capabilities, possibly with remote proctoring, at partner organizations offices because we think that will be a huge benefit for our partners.

We have a phase 2 transformation plan in draft which needs to be validated by all our internal stakeholders and we’re waiting for the CIC to validate it and make sure that we’ve got their priorities in there. But those are the sort of things that are in that plan to start in 2011.

John : A last but key question. If someone wants to make a career working with SAP, will taking a certification help them get a job?

Sue : The importance and significance of having certification is increasing all the time. We’re talking to customers and working with our partner organizations, to ensure customers and partners know the importance of certifying, and that our certification meets what they need.

I’d recommend anyone to get certified.

You have to see certification as a card in an entire hand of benefits that you can bring to a prospective client or customer. It’s not a guarantee to a job, but it’s a damn good card in the hand! If you’ve got experience, good communication skills, the necessary education background and you’ve got a certification, then if it’s like on like, they’re very likely to take you for the job, much more than the individual who has the same but without the certification. 

You can see more about SAP Certification at http://www.sap.com/services/education/certification. This is a second of a series of interviews with people making a difference to the SAP Ecosystem in learning and assessment. See Thought leader interview  – Prashanth Padmanabhan plans to revolutionize SAP’s learning and talent management for my previous interview with Prashanth Padmanabhan, on the future of SAP’s learning and talent management.