Skip to Content

So what’s this blog all about?

An (really) important part of IDM is the provisioning to other systems. And it’s a part that loves to keep us on our toes, as the amount of threads and blogs about this topic show. Through my time working with IDM the provisioning stopped working quite a few times due to different reasons. And every step of the way I learned a new reason and a new solution to get it going again. To keep up with all of that and to help me solve it faster when it decides to get stuck yet again I started to write down a checklist.

Some time ago I posted the better part of my little checklist in a thread and through some encouragement by Matt I decided to create a blog post out of it to share the whole thing to a wider audience and explain the steps a bit more. This is my first technical blog here on SCN so I’m a little nervous and excited at the same time. ^^

ℹ Just to be clear: Not all of the points of the checklist might work for you, since we’re on a Windows server with the IDM management console and use an Oracle database.

My tools

  • Access to the Management Console (MMC) of the IDM
  • Access to the Monitoring-tab via http://<portalurl:port>/idm/admin
  • SQL developer with database connection to IDM database
  • Permission to access the Windows services of the IDM-server and to start/stop the services
  • Permission to reboot the IDM-server
  • Really good connections to the database administrators

How do I know it’s that time again?

There are three signs that I check if the provisioning is really stuck again:

  1. I look at the “Job log” in the MMC to see if the last entry for the provisioning-dispatchers is from more that 15 – 20 minutes ago (even through it was triggered in the last minutes).
  2. The provisioning queue on http://<portalurl:port>/idm/admin is only growing.
  3. The dispatchers, that are assigned to do the provisioning, are shown as running in the MMC under “Dispatchers > Status” and the timestamp for “Last check” is updated when I click on refresh.

If all those steps come back with a “yes”, I’ll get…

The Checklist

  1. Check the “Status”-overview in the MMC to see, if a job is showing the state “Error”.
  2. Restart the provisioning dispatchers in the “Services”-menu of the server.
  3. Check for waiting tasks via the SQL developer.
  4. Check the “Windows RT conn. string” on the Database-tab of the Identity Center configuration in the MMC.
  5. Reboot the server the MMC and dispatchers are installed on.
  6. Restart the IDM database.

That’s the checklist in short, if you just need a little reminder or an idea for what to look at next. I’ll explain the points a bit more in detail now.

1. Check the “Status”-overview in the MMC to see, if a job is showing the state “Error”

In the MMC you’ll find the “Status”-overview as the first entry under “Management”.

ScreenShot089.jpg

It shows all the jobs for that Identity Center connection (in this case it’s named IMP). To check for jobs that have the current state “Error”, just click on the column header “State” and it will be sorted by content. If you have checked the box “Show only enabled jobs” at the bottom of the page, the jobs with error-state should be shown in red font at the top or end of that list.

If you find a job that is associated with provisioning and it’s on “Error”, right-click on it and start it with “Run now”.

2. Restart the provisioning dispatchers in the “Services”-menu of the server.

Go to “Start > Administrative Tools > Services” on the IDM server and look for your dispatchers, that are assigned to the provisioning, in that list. They should be shown as started. Right-click on them and choose “Restart”.

ℹ This is what gets our provisioning going most of the time.

3. Check for waiting tasks via the SQL developer.

Open the SQL developer and work your way through the following SQL statements:

select * from mxp_provision where msg like 'Wait for%'

This checks for tasks, that wait for the successful state of other tasks. The MSG-column gives you an auditid for the next SQL-statement. It’s the ID of the task, that is blocking the execution of the first task.

select * from mxp_audit where auditid=<auditid>

The MSG-column now shows information about the state (e.g. failed) of the blocking task, the reason and for which user and which assigment. With these information you can decide to leave it alone to handle itself (because it’s got nothing to do with the blockage) or you can use the next SQL-statement.

update mxp_audit set provstatus=1100 where auditid = <auditid>;

This last statement sets the blocking task to the state “OK” (= 1100) and therefor the waiting task (the one you found with the first statement) can finally be executed.

4. Check the “Windows RT conn. string” on the Database-tab of the Identity Center configuration in the MMC.

When you click on the IC configuration (the “IMP” in the first screenshot), the “Database”-tab will be displayed. At the end of it you’ll find the string under “Runtime engine”.

ScreenShot090.jpg

Open it and test the connection on the “Connection”-tab. If it comes back as failed, correct the name of the data source and/or the logon information. Then test it again to see if it’s successful now.

5. Reboot the server the MMC and dispatchers are installed on.

Well, that’s pretty self-explanatory. ^^

If you don’t have permission to do this yourself, have the contact data of the administrator(s) at hand, who can reboot the server for you (just like with the restart of the services in #2).

6. Restart the IDM database.

This was only necessary once (until now), but to complete my checklist I’ll include it here, too. Since I don’t have direct access to our oracle database, I let our database administrators handle this one for me.

What’s more to say?

Well, that’s it! I hope the list can help you when your provisioning decides to take a break again. This is – of course – nowhere near a complete list, but a result of my experience with the issue.

If you have some tips of your own to add, I absolutly welcome it! As you know “Sharing is caring”. 😉 So leave a comment when you have your own little checklist for this issue or if you want to give some feedback for my blog (which I’m really looking forward to, because I am a big fan of constructive criticism and it is my first technical blog).

Thank you for your time and attention. I hope it was not wasted! *g*

Regards,

Steffi.

To report this post you need to login first.

31 Comments

You must be Logged on to comment or reply to a post.

    1. Steffi Warnecke Post author

      Thank you so much, Tammy! 🙂

      Also for posting the first comment. The anticipation for that very first comment is probably the hardest thing to endure for anybody sharing content with the community. *g*

      (1) 
  1. Matt Pollicove

    Steffi,

    EXCELLENT! So very well done!  Great information, well organized! Next step is for some people to share other tips (I’ll take a look this afternoon through my notes) and then it should probably be converted to a document so that we can keep adding on the info!

    Matt

    (0) 
    1. Steffi Warnecke Post author

      Thank you very much, Matt! *beaming*

      I was thinking about making this a document in the first place, so that others can chime in directly, but since it became more a piece about my experience etc instead of a common thing that goes for everyone I decided for the blog as content type. 🙂

      But you’re right, if more tips are collected then a document is better. But I’d keep this blog, too, because it’s more detailed than would be fit for the document, I think. I’ll add my points though and would link to this blog for the “more details here”-hint. ^^

      (0) 
        1. Steffi Warnecke Post author

          You got to ask a SCN expert for that. 😀 Are there guidelines for this?

          I would suggest just creating a document with a table and columns like “Issue, Solution, Tools needed, Comments” and everybody can just add. Maybe add “IDM Version, Database”, too. 🙂

          But this only makes sense, when there are some more tips to share. ^^

          And the quotes… I’m a big fan of yours, as you know! I’d never steal your thunder. *g* I have to say, I’m not that good with finding the right ones, so maybe I’ll just stick to the play with words and see, where that leads me. *g*

          (0) 
  2. Matt Fraser

    Your first technical blog?  Really?  Couldn’t tell, it’s extremely well done, and just what other IdM administrators surely need.

    (0) 
  3. Gretchen Lindquist

    Congratulations on your first technical blog, Steffi! Very well done. We have another team that supports our IDM solution here, so it’s nothing I have to deal with, but if I had to, at least now I would know where to start. Thanks!

    Gretchen

    (0) 
  4. Steffi Warnecke Post author

    Thank you, Matt and Gretchen! It’s always great to see the own work appreciated. I’m quite stunned by all the positive feedback, although that’s what I was hoping for, if course. *g*

    @Matt: I have written some documentation and training materials for my work over the years (for my company), but this is the first time I share the outcome with such a big audience and the first time I did it in English. 🙂

    @Gretchen: You could point your IDM crew this way, maybe they can add to the list. ^^

    (0) 
    1. Matt Pollicove

      Steffi,

      Your English is perfect! I did not see any errors.

      You could have used a quote, don’t forget that I own the SF/Comic franchise for the IDM Forum! 🙂

      Matt

      (0) 
    1. Steffi Warnecke Post author

      Thank you, Lakshmi and you’re welcome. *g* I learned so much through the blogs and documents (and answers to questions) other members published here, that I’m more than happy to pay some of it back. 🙂

      (0) 
  5. Siarhei Pisarenka

    It was really the hell with provisioning in IDM 7.2 SP8. Dispatchers was not working longer than one hour then became frozen. The database was full of blocking queries. It was absolutely not possible to keep the queue in a stable state.

    Lots of the issues were resolved with patching IDM SP8 with the latest patches – we installed patch 5. I really recommend looking at the patches available for your version.

    (0) 
    1. Steffi Warnecke Post author

      Hello Siarhei,

      well, I did not encounter such extreme trouble with the provisioning in 7.2 SP8 like you described and I’m really thankful for that. ^^ Dispatchers stopping every hour etc… sounds like a horror movie for IDM-admins. 😀

      Nontheless, sometimes the provisioning is acting up in our system and the steps in the checklist helped me with it every time.

      (0) 
      1. Siarhei Pisarenka

        We use MS SQL database where the problem of locks (sometimes dead-locks) happens very often. Perhaps in Oracle DB there is no such issue, or it’s not so visible as in MS SQL. Due to locking of the core IDM table (semaphore) the dispatchers of the pure 7.2 SP8 often stop working silently.

        In the 7.2 SP patch they resolved some of these issues, so the dispatchers became to behave much more robustly.

        (0) 
        1. Steffi Warnecke Post author

          I think it was with SP7, that we had to watch the dispatchers this closely, since they loved to stop ca. once a week (really stopped this time, not just go into sleep-mode), but I’m not quite sure. But I know the problem. Just not hourly. So I’m pretty happy now, that we are on Oracle. 😀 And the dispatchers run fairly stable, too, now.

          (0) 
  6. Laurent Vandenbemden

    Hello Steffi,

    Wonderful blog, would be excellent to have such detailed procedures for the different “usual” issues we encounter with the provisioning in IDM. 🙂

    I have an additional question however about step 3 “checking the wait tasks with SQL developer”. Do you do an additional step to trigger the task execution after the provstatus has been updated to 1100?  This could maybe be something to add.

    Thanks and again very well done,

    Laurent

    (0) 
    1. Steffi Warnecke Post author

      Hello Laurent,

      thank you for the nice feedback!

      And no, there is no need. As the waiting task is checking (but please don’t ask me how 😀 ) for the other task to go to “OK”-state, it will start as soon as the last sql-statement (the update) is done. So no additional step is needed to trigger the execution of the first task.

      (0) 
      1. Laurent Vandenbemden

        Hey Steffi,

        Welcome 😉

        Hmm strange, In my case the provisioning remains idle for these “waiting entries”.

        The queue has already growed around 500 of these tasks in wait state. Would it be possible that I first need to update all of them before the task would start over? I started with updating the 20 oldest .

        Maybe I should open a sepparate tread for this?

        Thx a lot,

        Laurnet

        (0) 
          1. Laurent Vandenbemden

            I’ll open a separate thread but first I might have to apply the patch n°6. I can see that almost all my pending entries are associated to the modify tasks. So it might be the bug described in SAP Note 2026649.

            (0) 
  7. Lakshmi Vuppuluri

    Hello Steffi,

    We recently upgraded Identity Centre to 7.20.9. The current dispatcher version is 7.2.8.1 (MxDispatcher) How can I upgrade only IdM dispatcher service?

    Thanks

    Lakshmi

    (0) 
      1. Lakshmi Vuppuluri

        Tero – thank you for the response.

        Do we need to uninstall the current runtime components and then upgrade or an in place upgrade is possible?

        Regards, Lakshmi

        (0) 
        1. Tero Virta

          I’ve always uninstalled both runtime and design time before upgrading by installing the new one. Although I think the design time (MMC) is more crucial to uninstall due to the ActiveX/COM-components it uses (which have Windows registry references).

          regards, Tero

          (0) 
          1. Matt Pollicove

            Hi Tero,

            Uninstall/Reinstall has always been best practice for all IDM components.

            Of course make sure that the dispatchers and event agents are stopped as well!

            Regards,

            Matt

            (0) 
    1. Steffi Warnecke Post author

      Thanks, Hendrik.

      The more hands-on blogs and information we get about IDM, the better. Do the IDM 8 version, I’ll need it in the future for sure. 😉

       

      Regards,

      Steffi.

      (0) 

Leave a Reply