SAP Identity Management Provisioning Monitor Tool
We showed Lab Preview version of Provisioning Monitor tool at TechEd’2015 and since some time both Identity Management 7.2 and 8.0 customers can enjoy it because it is release independent component.
Based on information form support questions we identified several typical issues and the tool can recognize those and propose solutions for them.
The provisioning monitor can help you analyze and troubleshoot performance issues, such as:
- A task or а process are stuck in the provisioning queue
- Many entries are stuck in the provisioning queue
- The system is too slow, as the execution of tasks or processes is delayed due to a full provisioning queue
The tool reads data from database, it does not do any changes but can show you and advise you what might be the issue by showing the workflows in the provisioning queue, which are related to a specified entry (for example, person, privilege, or role). The output are two JSON files – Workflow.json and Problem.json.
While Workflow.json can give you insights about what is going on with the entry – the workflow as the name implies, the second one contains additional information about the problem and a proposal for a solution, if the analysis of the provisioning monitor has detected one or more of the following root causes:
- Tasks are stuck in state „Queued“ but there is no task which they are waiting for to finish
- Tasks are stuck in state “Ready to run” with no execution time
- An object references itself and can’t be completed
- Tasks are stuck in state “Wait for event tasks” or “Wait for event task status”
- Approval tasks without assigned approvers are stuck in the provisioning queue
- There are issues with the dispatcher configuration of the system
Let’s look in more details.
1. Tasks are stuck in state „Queued“
The problem is that there are tasks stuck in state “Queued” (State = 37) in the provisioning queue which would never be executed because there is no task which they are waiting to finish or all the tasks are completed.
Solution : The task state in the provisioning queue have to be updated to 2 – “Ready To Run”.
Additional Information : You can fix such problem with the following database statement :
SQL Server/ Sybase ASE – UPDATE mxp_provision set State = 2, PrevState = State where State = 37 AND ActionID = <ReportedProblemTaskID> AND MSKey = (select mcmskey from mxi_entry with (nolock) where mcmskeyvalue = <EntryMskeyvalue>)
Oracle/DB2 – UPDATE mxp_provision set State = 2 where State = 37, PrevState = State AND ActionID = <ReportedProblemTaskID> AND MSKey = (select mcmskey from mxi_entry where mcmskeyvalue = <EntryMskeyvalue>)
2. Tasks are stuck in state “Ready to run” with no execution time
The problem is that in the provisioning queue exist tasks which are in status “Ready To Run”, but there is no scheduled execution time about them.
Solution : The exectime in the provisioning queue has to be updated.
Additional Information : You can fix such problem with the following database statement :
- SQL Server/ Sybase ASE – UPDATE mxp_provision set ExecTime = getdate() where State = 2 AND ActionID = <ReportedProblemTaskID> AND MSKey = (select mcmskey from mxi_entry with (nolock) where mcmskeyvalue = <EntryMskeyvalue>)
- Oracle/DB2 – UPDATE mxp_provision set ExecTime = sysdate where State = 2 AND ActionID = <ReportedProblemTaskID> AND MSKey = (select mcmskey from mxi_entry where mcmskeyvalue = <EntryMskeyvalue>)
3. An object references itself and can’t be completed
The problem is that somewhere in a given task/process flow a task/process has a reference to itself.
Solution : The duplicated task/process has to be removed from the flow.
4. Tasks are stuck in state “Wait for event tasks” or “Wait for event task status”
The problem is that there are tasks in state “Wait for event task” (State = 23) or “Wait for event task status” (State = 24) in the provisioning queue which are waiting for event task which is already completed.
Solution : The status has to be updated respectively to 33 – “Ready To Process Wait Before task” or 34 – “Ready To Process Wait After task”
Additional Information : You can fix such problem with the following database statement :
SQL Server/ Sybase ASE – UPDATE mxp_provision set State = State + 10, PrevState = State where State = <ReportedProblemState> AND ActionID = <ReportedProblemTaskID> AND MSKey = (select mcmskey from mxi_entry with (nolock) where mcmskeyvalue = <EntryMskeyvalue>)
Oracle/DB2 – UPDATE mxp_provision set State = State + 10, PrevState = State where State = <ReportedProblemState> AND ActionID = <ReportedProblemTaskID> AND MSKey = (select mcmskey from mxi_entry where mcmskeyvalue = <EntryMskeyvalue>)
5. Approval tasks without assigned approvers are stuck in the provisioning queue
The problem is that an approval task is stuck in the provisioning queue without approvers.
Solution : If the task is of type Basic Approval, the approvers which are assigned to the task have to be checked. If there are assigned approvers they have to check their “To Do” tab in the Identity Management User Interface. If no approver is assigned, assign an approver and trigger the approval task again.(One way to do it, is to set the task to be triggered on modify in the Event Tasks property of the attribute. And then modify the attribute)
If the task is of type Assignment Approval, the Development Support has to be contacted.
More information on how to get provisioning monitor tool for 7.2 and 8.0 SP1 and how to use it you can find here:
The difference between 7.2 and 8.0 is that in 7.2 it is a command-line standalone tool
while in 8.0 it is integrated in the Monitoring tab of administrative UI.
Here is an example of how to call it in 7.2. in Windows environment:
ProvMonitor.bat “jdbc:sqlserver://localhost:1433;databasename=MXMC_db;user=MXMC_oper;password=_Password” “C:\usr\sap\idm\JDBC Drivers\MSSQL JDBC40\sqljdbc4.jar” “mskeyvalue.demo” “1” “DEBUG”
You can put this string in a shell file and if you do not want to put the password inside use “_Password” and you will be prompted during execution.
We are glad to receive feedback about the tool or other frequent patterns you may have faced.
Hello Fedya,
this sounds pretty interesting, but the note you referred to is not released yet. 🙁
Is the link you provided a SAP internal link? I get a nice proxy error message when following it and had to look for the note through the support portal.
Regards,
Steffi.
Hi Steffi,
I added also in the blog- this is a pilot note so you have to request to be added as a pilot cusotmer from our primary support.
Meanwhile we will provide additional simpler delivery channel.
Best wishes,
Fedya
Hi Steffi - once you are on 8.0 Sp2 you can call it from the webui. As Freda stated if you have a prov queue issue then open a ticket and we can provide the tool and add your organization to the note which details how to execute it in standalone mode.
Cheers,
Chris
This looks very interesting.
How do I request permission to view the note? How do I access the primary support?
Hello,
the prov monitor is documented in the IdM 8.0 online help and can be downloaded now from the link SAP Development Tools
website.
regards,
Chris
I tried clicking on the link above to get this tool and it keep saying that the page cannot be displayed. Is that a current link?
We're always dealing with stuck jobs in my environment so this tool could come in really handy. Please advise me on how I might get it. Thanks.
Never mind. Upon further review, I see Chris' comment linking to the tool. Thanks.
What exactly is the MSKEYVALUE operator looking for here? Is there supposed to be an actual MSKEYVALUE in here, a list of them in a text file, and what MSKEYVALUEs are being asked for? The MSKEYVALUEs of the stuck jobs?
Please advise on how to properly plug the right stuff into the command line execution.
I'm guessing any MSKEYVALUE,
OK. I guess I can try your approach below and just set MSKEYVALUE to Administrator and see what happens. I just wonder what the purpose is of having that switch on the command line if it doesn't mean anything.
Hi Brandon,
The MSKEYVALUE is the one one the user for example which has problem with the provisioning and you want to troubleshoot.
Best wishes,
Fedya
OK. So I would just put in the MSKEYVALUE of my IDM account in that spot? That sounds simple enough. What is the reason for needing this information? Wouldn't the same report come out no matter who's running the tool? Just being curious. 🙂
Hi Brandon,
This is not the MSKEYVALUE of your IDM account.
Normally you would know the MSKEYVALUE of the entry which is stuck and that's what you put there.
best wishes,
Fedya
Hi Brandon - see the link here for the instructions
Running the Provisioning Monitor from the Command Prompt - SAP Identity Management Solution Operation Guide - SAP Librar…
and when calling the command ensure to put quotes " between each parameter e.g.
"<replace with JDBC_URL>" "<replace with JDBC_DRIVER_PATH>" etc.
cheers
Hi Chris - very useful to know about the quotes - any chance the help file could be updated with this?
Thanks!
Hi Andy - thanks - this is already an action item for our documentation team.
Regards,
Chris
Hi Fedya, I am trying this with 7.2 SP9 on SQL Server 2012 . I've hard-coded the parameters into the batch file, but I get the message: databasename=mxmc_db was unexpected at this time.
Any ideas?
Here's my ProvMonitor.bat
@echo off
:: =====================================================================================================================================
:: --- The Provision Monitor tool needs a path to JDBC driver and JDBC URL to connect to the Identity Management database.
:: --- You can also specify the log level of Provision Monitor logger. It must be one of ("trace", "debug", "info", "warn", or "error").
:: =====================================================================================================================================
SET JDBC_URL=jdbc:sqlserver://192.168.253.71:1433;databasename=mxmc_db;user=mxmc_oper;password=NOTTELLINGYOU
SET JDBC_DRIVER=C:\JAVALIB\sqljdbc4.jar
SET MSKEYVALUE=Administrator
SET IDSTORE_ID=1
SET LOG_LEVEL=DEBUG
IF [%JDBC_URL%]==[""] goto USAGE
IF [%JDBC_DRIVER%]==[""] goto USAGE
IF [%MSKEYVALUE%] == [""] goto USAGE
IF [%IDSTORE_ID%] == [""] goto USAGE
IF NOT [%JDBC_URL:_Password=%]==[%JDBC_URL%] (
echo Please enter the password:
MXGetPwd.exe > pwd.tmp
set /p JDBC_PASSWORD= < pwd.tmp
setlocal ENABLEDELAYEDEXPANSION
set JDBC_URL=%JDBC_URL:_Password=!JDBC_PASSWORD!%
del pwd.tmp
)
SET RECURSION_DEPTH_LEVEL="10"
"%JAVA_HOME%\bin\java" -cp lib\*;%JDBC_DRIVER% -Dorg.slf4j.simpleLogger.defaultLogLevel=%LOG_LEVEL% com.sap.idm.monitoring.ProvisioningMonitorMain %JDBC_URL% %MSKEYVALUE% %IDSTORE_ID% %RECURSION_DEPTH_LEVEL%
goto EXIT
:USAGE
ECHO Please insert correct parameter values.
ECHO.
ECHO The Provision Monitor tool needs a path to JDBC driver and JDBC URL to connect to the Identity Management database.
ECHO In order to run the Provision Monitor tool for a given entry in identity store please follow the syntax:
ECHO.
ECHO ProvMonitor.bat "JDBC_URL" "JDBC_DRIVER_PATH" "MSKEYVALUE" "IDSTORE_ID" "LOG_LEVEL"
ECHO.
ECHO The log level of Provision Monitor logger must be one of ("trace", "debug", "info", "warn", or "error").
ECHO If you do not specify or pass wrong log level it will be considered as "info".
goto EXIT
:EXIT
Thanks,
Matt
I was able to get it working. JAVA_HOME was not correctly populated.
Now to use it and get some results 🙂
Anxious to hear the results.
Hi Fedya,
Thanks for the document.
We have some pending tasks could not be executed in mxp_provision table. but the status is 42 (Grouping. Waiting for assignments) instead of 37.
Do you have any idea how we can re-trigger the provision?
Thanks,
Chenyang
Hi
this is also working for state 42.
Thanks for the article.
Cheers
Chenyang
I have updated the download site for the tool:SAP Development Tools
Hi Fedya
I have entries that are stuck in state 33. The "wait for event..." audit id has the provstatus 1000.
How do I clear these please.
Thanks
Ranjit