Skip to Content

Interview by John Kleeman, Questionmark Chairman and assessment expert, with Sue Martin from SAP, on how she and colleagues are making the SAP Certification Programme a key enabler for successful SAP projects. Why certification matters to SAP, SAP customers and SAP specialists.

 

John : Sue, what is your job role at SAP?Photo

Sue : I am Global Certification Manager, so I’m responsible for the Certification Programme and Strategy at SAP worldwide. I work together with other teams within SAP to ensure that we get the exams out, we’ve got the right strategy in place and that the certifications focus on the right priorities.

 

John : How did you come to join SAP?

Sue : I originally came to SAP four years ago as a freelance consultant. SAP’s certification programme had been around since the mid-90s, but it hadn’t changed a lot in the previous 10 years. I was asked to review and identify ways we could enhance the programme.

The market for SAP had changed immensely, and the certification programme hadn’t really changed to match the market needs. So I did an analysis and review, spoke with a lot of stakeholders and customers, and made 10 recommendations of how we should change the certification programme. In a nutshell the change was moving from certification as a post training assessment, which validated that you knew what you’d learned in classroom courses, towards something that provides value to the individual and a benchmark on which customers can base important decisions.

After I’d done my review, the majority of the recommendations were accepted, and that was when I was asked to come aboard to SAP permanently to implement it. And that’s what we’ve been doing for the last 3 years.

 

John : What is SAP’s motivation, is to make money from certification as a revenue stream or to make SAP projects and customers more successful?

Sue : Only the second!

The post training assessment side of things was proving the return on investment in education, but we needed to take a step back and look at the big picture. Certification should not be there just to comply with educational objectives, but it should have a higher mission, to drive quality into the market, to protect our customers and ensure their success with SAP software. That has been the basis of the transformation.

 

John : If you are an SAP customer, what is the benefit of recruiting people who have a certification?

Sue : One of the points of the transformation was that we took it beyond just the new hire certification. The old certification exams were very much aimed right at the beginning of your SAP career.

But we’ve changed the exams to be Job Task Analysis based, where certifications test what you need to know in the job.  It’s not just about knowing functions and features of SAP software, but it’s about what can you actually do, because this gives the customer more value.

As a customer, I’m not interested in bringing someone on board because he or she knows everything you ever wanted to know about SAP software plus lots that you don’t really need. I’m more interested as a customer in knowing exactly what that individual can do, what tasks can they perform, how can they help me ensure business success.

We also aligned the job roles being certified with needs, and validated this with partners and customers. We’ve made certifications more scenario-based, gone away a little from “which button do I press” to “how would I address that particular customer concern” or “how do I solve a particular customer issue”. And that adds immense value to me as a customer, to know that someone understands my business needs and can interpret those into software requirements.

The second step that enhanced the value for customers is that we’ve added the second, Professional layer to certifications. These are aimed at validating people with 4+ years of experience, who have got real business experience. We’ve now got Professional certifications in more or less every area. These are very much scenario based questions, all the questions are written by higher level consultants, senior consultants who are in the job, doing the job, faced with those business challenges every day. It’s not just knowledge recall or something you could learn out of a book. Bringing that level in, we’ve now had our first couple of thousand people running through the Professional certification programme, and I think that makes a huge difference.

 

John : What effort goes in at SAP to make the questions good quality and set the scoring fairly?

Sue : We follow industry standard practices to get legally defensible exams into place.  In order to get more experiential subject matter expertise, we moved from instructors writing questions to much more involvement from our own consulting organization. At the same time, to get the level of consistency and quality of review, we significantly increased the number of subject matter experts (SMEs) involved.

Here is an overview of the steps we follow:

  1. The first step is job task analysis to identify the tasks that we ask questions on. These are validated with customer organizations, partners and with our own consulting organization.
  2. The next step is the design phase, where we have the item writing workshops that write and review the questions. It depends on the subject matter, but there are normally at least 6 or 7 SMEs that sit in these workshops, supported by a psychometrician.
  3. We then have peer review from another set of SMEs after the workshop. By SMEs I mean primarily people who are actually working in that area, so for a Professional certification that would be a Senior consultant or a Platinum consultant.
  4. Then another group of SMEs go through the exam as beta testers and we look at the results.
  5. Then a group of experts goes through the questions and do a pre-standard setting process, estimating the difficulty level of each question.
  6. Then we have the final standard setting meeting to get to the cut score (pass score) determination phase.  And then at the end of all that, we finally have the exam!

 

John : I know you use Questionmark software in authoring and delivery of exams. Is our Questionmark technology working well for you?

Sue : One of the important parts of the transformation phase 1 was to be more flexible from a delivery perspective and with authoring tools. So we decided in 2007 to move over to an industry standard system (industry standard being the key to everything that we are trying to do in this project) rather than our previous in-house developed test delivery and creation tool. That was when we migrated over to Questionmark software.

We rolled out Questionmark Perception to over 80 SAP education centres worldwide and that now provides us with excellent content authoring tools which really enable us to author effectively, working with a large number of subject matter experts. And it allows us to have a great delivery tool for all of our SAP education centres. We also have in place test delivery via Pearson Vue centres around the world to increase accessibility throughout our Ecosystem.

 

John : Can you share how many people a year get certified? How many Professional and Associate?

Sue : We’ve got about 250,000 certified individuals around the world. We’re currently running at about 60,000 people a year getting certified.

The Professionals are growing hugely. We’ve got the first 2,000 Professionals, and a lot of that has been in the last 6 months to a year. So it’s quite a steep curve.

 

John : Do SAP employees themselves get certified?

Sue : Some time ago, experienced people used to say that they didn’t need certification, as they already had experience. When we transformed the certification, we said this was unacceptable – how could we as SAP go to the outside world without being certified ourselves, and say this is a credible programme?

So we sat down with our consulting organization and took in their feedback, and ensured we were working closely with them to ensure that our programme was really addressing their needs. And that in itself has been really good as there have been quite a few emerging roles that we’ve put into our certification programme that have come from our consulting organization. The business of technology evolves, so the collaboration with our consulting organization has been a win-win, it’s also meant that we’ve got 95% of our consulting organization certified in the last couple of years.

 

John : Do you have to take a specific training course before taking a certification, do some certifications require SAP or SAP partner training?

Sue : From a global perspective, no we don’t, there is no pre-requisite to take a training course first.  On-the-job experience can be equivalent to training.

We do have security issues in certain geographies, and in two countries in Asia, we do require training prior to certification as a security check to prevent inappropriate test takers. 

 

John :  You’ve been talking about transformation phase 1, what is phase 2?

Sue : Our aim for phase 2 is to make sure we are lifting the programme up to a different level, and that it’s really going to improve the quality of customer implementations. One of the things we are bringing into place in February is a Certification Influence Council (CIC) to help us make phase 2 of the transformation happen. This has been instigated by Bill McDermott our co-CEO and will have internal stakeholders from all parts of SAP and external stakeholders.

As well as internal stakeholders, the CIC will have representatives from user groups, partners, customers – and also two of the “certification 5”, industry analysts who’ve blogged about certification.  They will be meeting every quarter to review our direction and ensure that certification is really improving business success as a customer.

 

John :  What improvements are likely in phase 2?

Sue : Some of the things we are looking at: we want to create and nurture a certified community, so we need to have more benefits in place really for the individual, we are considering more flexible virtual learning opportunities for the individual and also more benefits for the certified community – access to events, access to peer exchange, more privileges in terms of discounts off our press books.

Then we’re looking at ways to improve the certification. One area to look at is live application testing, and we’ll be working with the CIC to see whether there are workable and scalable models we can use. We’re also looking at the Master level, which will be aimed at the crème de la crème, the real experts, who have 8+ years of experience. Now that the Professional level has got some l recognition in the market, we’ll be looking more closely whether or when we’ll introduce the Master level.

Then we will look at other exam development techniques, and I think Questionmark can help us with that, things like essay marking and more innovative item types. Also one of the things on our agenda is ways to provide testing capabilities, possibly with remote proctoring, at partner organizations offices because we think that will be a huge benefit for our partners.

We have a phase 2 transformation plan in draft which needs to be validated by all our internal stakeholders and we’re waiting for the CIC to validate it and make sure that we’ve got their priorities in there. But those are the sort of things that are in that plan to start in 2011.

 

John : A last but key question. If someone wants to make a career working with SAP, will taking a certification help them get a job?

Sue : The importance and significance of having certification is increasing all the time. We’re talking to customers and working with our partner organizations, to ensure customers and partners know the importance of certifying, and that our certification meets what they need.

I’d recommend anyone to get certified.

You have to see certification as a card in an entire hand of benefits that you can bring to a prospective client or customer. It’s not a guarantee to a job, but it’s a damn good card in the hand! If you’ve got experience, good communication skills, the necessary education background and you’ve got a certification, then if it’s like on like, they’re very likely to take you for the job, much more than the individual who has the same but without the certification. 

 

You can see more about SAP Certification at http://www.sap.com/services/education/certification. This is a second of a series of interviews with people making a difference to the SAP Ecosystem in learning and assessment. See Thought leader interview  – Prashanth Padmanabhan plans to revolutionize SAP’s learning and talent management for my previous interview with Prashanth Padmanabhan, on the future of SAP’s learning and talent management.

 

To report this post you need to login first.

11 Comments

You must be Logged on to comment or reply to a post.

  1. Mark Förster
    In the last decade I have taken several SAP certification tests. The single most annoying “feature” were ambigous questions. Of course every now and then you encounter questions you simply cannot answer, that’s normal. But if you encounter a question which is ambigously fomulated, then you have a really hard time during the certification test, even if you are very confident about the topic!

    If the one who wrote the question had this in mind, then of course the correct answer is “A”.
    But if the one who wrote the question had that in mind, then of course the correct answer is “B”.

    There are LOTS of such questions in SAP certification tests, and it is not just me who is annoyed by such questions (it is not just me who thinks the question isn’t stated clearly).

    (0) 
    1. Susan Martin
      Hi Mark,
      thanks for the feedback! That is indeed one of the issues we set out to address and we have found that the new processes we have introduced over the last couple of years – item writing workshops with a high number of consultants, peer reviews with other subject matter experts and pilot testing with another group as part of the standard setting process  and more extensive language edits – have made a substantial difference to the number of appeals we get on ambiguity.
      Sue
      (0) 
  2. Arun Bala
    Hi team, Would love if SAP Education provided some transparent methods/policies for SAP Certification in India. This is reg. the certifications booking/pre-scheduling options as followed in the other geographies. As of now there’s no options to book slots & pursue certifications. Plus, would strongly believe a great enthusiastic response would be given for this. Thanks.
    (0) 
    1. Maggie Ramaiah
      Hi Arun,

      Since India has very high number of certification takers and also with different backgrounds therefore the methods and policies are explained to each of them in detail.  No one policy appies to all candidates hence you dont see them on the portal.  However if you are interested in the certification, you can approach our office and the concerned person can give you the required information.  You can contact joyce.immaculate.priya@sap.com for certification related queries.

      Regards,
      Maggie

      (0) 
  3. Tom Cenens
    Hello

    I’m looking forward to the changes which will come in the future for the certification program.

    I’ve done 3 certifications in the last 4 years but I don’t see them as a card saying “I can this work”, the writing on the card is more “I have the knowledge of these courses”. So I’m glad to see that things are shifting in a direction that the certificate can get more value.

    Only coloring A,B or C to get certified isn’t enough in my opinion. The higher levels should have an even greater involvement of other methods (paper, interview, ideas).

    Why not put masters at work to give their opinion on a way to improve the area or product they are certifying for. When you have that much experience on the topic or product, should you not be able to identify area’s of improvement?

    I do have some concern on the years of experience that is being pasted on the different levels. How do you measure that?

    I mean someone who has been working in SAP for 10 years might know less about a specific topic (perhaps because he spend his time on lots of different topics or areas) than someone who has been around for 4 years or less spending almost all his time on one topic or specific area.

    Why not give someone with 4 years of experience (like me) the chance to go for a master if he feels up to up.

    I really enjoyed watching the following video on “changing education paradigms”. Besides it being a great topic, the video also has a marveleous way of displaying and I recommend it if you haven’t seen it:
    http://www.ted.com/talks/ken_robinson_changing_education_paradigms.html?awesm=on.ted.com_8qIE

    Kind regards

    Tom

    (0) 
  4. Andy Klee
    Impressive interview, John and Sue.  Well constructed arguments in favor of certification.

    Over time (probably the next five years), we’ll probably see fewer experienced SAP consultants arguing that certification is meaningless–if the next phase of SAP’s program plays out. 

    More experienced consultants will sit for the Professional and Master level exams because the tide will gradually shift towards certification as a valid, but not the sole, criterion to ensure successful implementations. 

    I recently sat for one of Oracle’s JD Edwards certification exams, and I found the whole process to be less than satisfying–almost no enhancements in the last 10 years were tested, and at the end Oracle actually asked test takers if we felt the exam was a relevant measure of our ability to implement the software. (That’s obviously the most critical aspect of any certification program, and to ask test takers what we think indicates that Oracle isn’t really doing their own psychometric research on the effectiveness of their exams.)

    (0) 
  5. NICOLA MC DONNELL
    I am very supportive of the new strategy concerning the SAP certification programme, and would like to congratulate Sue for a clear vision in this important field.

    The Phase 2 programme improvements that have been outlined in the interview are certainly welcomed by the SAP certified community, of which I am a member. I accordingly look forward to seeing how these unfold in due course.

    On a personal note, I wanted to identify one related aspect that I feel should be on SAP’s radar, especially given SAP’s strategy to provide more flexible learning opportunities with phase 2 of the certification programme (of which I am very supportive, since currently only a very small number of flexible learning style options are available, depending on your location). The matter relates to the online learning solutions, and specifically the eAcademies. The quality of these on-line course offerings should also be of a high standard (just as you would expect in a classroom environment) in order to ensure a good investment in education and be beneficial both to the individual and also to protect customers. I totally agree that SAP certification should have the right focus, and would simply like to emphasise as part of SAP’s overall learning and certification strategy, that an equal degree of rigour and scrutiny be applied to the content of flexible learning solutions before they are released mainstream in order to ‘raise the bar’ from a quality perspective. I was happily impressed by the steps outlined in the interview that pertain to how the quality of the certification questions are ensured, and it would be great to see the same consistency and quality reflected in the elearning material preparation in due course.

    As a senior business analyst with 6.5 yrs of SAP HCM experience, one of the challenges I currently face here in SE France while searching for a new opportunity (despite my extensive experience, my strong educational background, my excellent communication skills, and recent SAP certification under my belt) is that it is extremely difficult to ‘break into’ the SAP consulting world even as a junior consultant.  This challenge could be purely the result of my geographic location and the market conditions locally, however in theory the associate level qualification is designed to confirm/endorse the level of junior consultant. Yet employers here insist on actual work experience as a junior consultant.

    It would be excellent if at some point SAP would investigate expanding it’s course / certification offerings at the associate level  (e.g. SAP HCM eAcademy and others) to introduce follow-on courses that cover off in more depth more of the consultant bread & butter type activities like configuration, that customer’s desperately want their hires to have practical experience of, in order to bridge the gap. A further range of practical / hands-on system courses that build on the introductory offerings at associate level and which address those day to day tasks that a SAP junior consultant would most certainly be asked to perform in a given job role e.g. HCM junior consultant, would definitely boost a SAP junior consultant’s chances of employment, if these SAP courses / certifications are deemed to be aligned to the underlying business needs customers have in these areas and therefore valuable. Just as the SAP certification programme emphasis has shifted to exams being more Job Task Analysis based, so too should the SAP course offerings be equally as practical in their outlook in order to give the individual a chance to make their investment in SAP education payoff and also to give the customer more value and confidence in the resources that they are hiring.

    (0) 
  6. Jon Reed
    As part of the Certification Five, I see certification issues differently than SAP in many respects but there is also common ground. Since my own views and that of the C5 are well documented I’m not going to re-iterate them here.

    I would like, however, to make a few points:

    – Bill McDermott driving this initiative is vitally important and the C5 is very encourage that he continued to be receptive to making SAP certification an industry leader.

    – Eventually I hope to see more financial investment in certification by SAP and believe it will be necessary for SAP to make that investment in order to make SAP certification a true benchmark leader.

    – The establishment of the Certification Influence Council (CIC) is a crucially important development. The C5 will be participating. My one correction to the comment by Sue as it is written in this blog is that the C5 are not just industry bloggers, we have three senior SAP practitioners in our midst as well. We do our best to represent the views of the SAP community at large on certification and regularly gather field data and community viewpoints to keep ourselves on track.

    – The Master level was referred to somewhat hypothetically in this interview, may be different if the discussion explored it further, I don’t know. I hope that it becomes more than hypothetical and I am looking forward to hearing the results of the pilot.

    – The C5 believes the Professional level while on the right track needs to move beyond pure multiple choice and we’ll be strongly advocating that on the CIC.

    – The presence of all the stakeholders on the CIC is a great development, as accountability to certification amongst consultancies can be tricky, and customers are crucial as it comes down in the end to what helps customers make better hiring choices.

    – Sue Martin deserves commendation for making the most consistent effort to dialogue with the community on certification. Sue continues to communicate with the C5 and even when we are in disagreement, or perhaps when we could have said things with more tact, she is gracious and keeping her eyes on the real prize, which is certification that individuals can aspire to and SAP customers can trust. She has my respect for that and I hope all who are interested in this issue take her efforts seriously and listen to her viewpoints.

    – Jon

    (0) 
    1. Susan Martin
      Thanks you Jon!
      Apologies if I did not emphasize the contribution of the C5 in this initiative adequately. Several  of the components of this transformation are a direct result of the hugely constructive dialog with the C5 group. I would also like to note that all of the C5 Mentors invested immense (voluntary and unpaid) effort into gathering and relating feedback to SAP last year with the Certification White Paper. We are incredibly grateful for this and look forward to continuing that dialog in the framework of the CIC.
      Jon  – the reason I am somewhat unspecific about the Master Level is because I would like to give the stakeholders of the CIC an opportunity to help us fix the priorities on the transformation over the next few years. We have a plan, which includes a timely launch of the Master level, but, as with most of the points on the plan, we would like to work together with the CIC stakeholders to ensure we are continuing in a direction that best reflects their business needs and to get their commitment. That is why I was a bit vague – I try not to do “hypothetical” if it can be avoided 🙂
      Sue
      (0) 
  7. Fabio Pagoti
    Hi,

    I really liked this post as it covers a topic I’m very interested in.

    However, I must disagree with the question related with pre-requisites when applying to a certification exam.

    Here in Brazil, I have contacted SAP quite a few times to get information about an ABAP certification. The answer was always the same:
    “It’s a pre-requisite to take a training course in SAP or in an approved partner before take the exam”.

    So: no course, no certification.

    I’ve been working with ABAP for 2 years. I already order books from SAP Press and read them several times. I agree that on-the-job experience can be equivalent to training and do have some but I must say that it’s NOT what happens.

    The company I work implemented SAP more than 10 years ago. All the training was done with who worked in that time. Now the company can teach new employees for itself. Why should the company pay 5 to 6 salaries in a SAP training course for a new employee if it can handle it locally?

    What should I do to prove I have experience to take a certification exam?
    Why do I always hear that I should do a course before take the exam?

    Please, I really would like to have these points addressed with Sue or other responsible.

    Regards,

    (0) 

Leave a Reply