For those who are managing BusinessObjects Business Intelligence Platform(BIP) as an Administrator, this article will provide some guidance for how to troubleshoot some common issues.
CentralManagementServer(CMS) can't be started
- Check from OS level log so to see if there is potential CMS related error.
- For linux, check the /var/log/messages to see if there is any boe_cmsd related errors. You will need to use linux root account to check the messages log. You may use the command: ‘cat /var/log/messages | grep boe_cmsd’.
- For Windows, check the Event Viewer -> Windows logs -> Application to see if there is any error/warning related with cms.
- If there is DB related error, involve DBA to resolve that.
- If no error from OS log, try to start CMS in console mode (KBA 1828536 for Windows and KBA 1846264 for Linux) to see if it is possible. If console mode can’t be started, try to repair the installation.
- If CMS can be started in console mode, then try to recreate SIA or create new SIA with this console mode CMS – choose to use existing CMS.
- Sometimes CMS performance issue can also lead to the CMS startup issue, check next section for investigation steps.
CMS Performance/Hung issue
- If the slowness happens during logon, check if the performance is good to load the logon page by accessing /BOE/BI or /BOE/BI/logonNoSso.jsp. This is to tell if issue caused by web tier.
- If loading the logon page is fast, but slowness happens at logon, then try to use Enterprise authentication with Administrator account and see if there is performance issue. This is to tell if issue is related with 3rd party authentication. CMS medium level log can also be used to confirm if the slowness is caused by 3rd party authentication, see below sample logs – the AD logon costs almost 55 secs:
2024 01 10 04:52:31:897|BIP::CMS:Logon
2024 01 10 04:52:31:908|WINAD: CKerbSessionMgr::GenServerContext() -- User: CN=xxxx
2024 01 10 04:52:50:057|WINAD: CADImpl::GetName() -------------------------------------------
2024 01 10 04:53:08:168|WINAD: CAliasId::CAliasId -- info extracted from alias xxxx
2024 01 10 04:53:08:169|WINAD: CADImpl::GetUserDesc() --------------------------------------
2024 01 10 04:53:26:295|WINAD: CAliasId::CAliasId -- info extracted from alias xxxx
2024 01 10 04:53:26:295|WINAD: ADAccountFactory::GetAccount() called for aliasId xxx
2024 01 10 04:53:26:350|BIP::CMS:Logon: 54.453 - If the slowness also happens when doing actions like navigation between different folders, logon CMC->Servers and check below CMS metric values:
Average Commit Response Time Since Startup (msec)
Average Query Response Time Since Startup (msec)
These two values are expected to be less than 50. You may feel overall performance slowness if these two values are larger than 200. These two parameters indicate the CMS DB performance so the larger value it means the worse CMS DB performance.
There are several possible reasons to cause CMS DB performance issue:- There are too many objects in cms_infoobjects7 table. This can be identified by checking the ‘Number of Objects in CMS System DB’ value from CMS metrics. Some possible reasons to cause the high number of objects:
- These objects could be instance objects, you can check ‘Completed Jobs’ value from CMS metrics or utilize BI Landscape report(KBA 2138275) to check the Content analysis part. If this is the case, you can reduce the instance objects number to use the Limit feature (KBA 1461109)
- There objects are platform search objects. This needs to check from BI Landscape report(KBA 2138275) -> Platform Search analysis to see if there are a lot of objects in each Queue. There is a known product bug for Platform Search Queue, see NOTE 2848886. You will need to upgrade to fixed version or manually delete the objects manually from CMS DB, see KBA 2448628. Please backup your CMS DB before any action.
- For other kind of objects, you can run below query to get the number of each object type:
SELECT typeid,count(ObjectID) FROM CMS_InfoObjects7 group by typeid
You can send this query result to SAP Support team so they can help check what are these objects.
- There are deadlocks in CMS DB. This need to be checked by DBA. Potential reasons to cause deadlock are:
- CMS DB statistics is not up to date, check with DBA for this.
- The CMS related tables are missing primary key or index, see KBA 2591142
- Platform Search is causing issue, see KBA 1640934
- SAP product bug, see NOTE 3333003
- The CMS DB itself is running slow, eg:
- Lack of enough hardware resource, eg: CPU, Memory, IO, etc.
- There are some other applications together with CMS to share the DB resource – we should have a dedicated DB instance for CMS DB.
- The DB itself is running some background jobs and these jobs could have been stuck or hang there.
- CMS DB should be in the same network/data center as the BIP host. If it is Oracle, you can try to use tnsping command to ping the CMS DB from BIP host and the result is expected less than 10 msec.
- (Very rare) CMS DB can’t provide enough available connection as CMS requested. Check the CMS metrics for below values:
Established System Database Connections
Currently Used System Database Connections
- Check from Auditing folder to see if there are many non 0 size pending files. This indicates there is issue with Auditing and this issue could cause performance issue.
- You have been placing Audit DB and CMS DB in the same DB instance/schema, CMS performance can be affected when there is issue with Auditing(eg: too many records in Auditing DB, or issue when doing Auditing housekeeping). In this case, try to turn off Auditing to see if performance can be better.
- Other possible solutions, see KBA 2232733
However, if the DB performance is good, you will need to check if there is enough hardware resource for the BIP host machine, eg: if there is enough free CPU and Memory, and also check if the BIP host IO speed is fast enough. You may logon the BIP host and try some OS level actions, eg: navigating between different folders, copying files, or running some applications, etc. to see if it is slow or fast thus to understand if the slowness if from the OS level.
Antivirus software or security software could also cause some weird issue. So disabling them could be an option if all above steps can’t help resolve the issue.
Scheduling issues
- If the jobs are stuck in pending status
- Check from the job server metrics to see the current job server workload matric and children number to understand the current workload. If the job server is already running with max workload, then check the job server children section to see if the children can receive new request. If the children are able receive and process new request, it means the job server is running fine. The pending issue is just because there are many jobs to be processed and there is a large pending queue. You can consider to increase the max concurrent job number to resolve this issue. And please also make sure there is enough free hardware resource when you want to increase the max concurrent job number.
- If there is few workload, or the children never receive new request, this could mean there is issue with current job server. Try to restart the job server to see if issue can be resolved. Please make sure to stop the job server first, then check from OS level to make sure there is no zombie jobserver/jobserverchild process running before you start it again. If the pending issue often happens in this scenario, you can consider to set the “Maximum Child Requests:” to 1 for the related scheduling service. There is also an existing bug for the pending issue, see NOTE 2948610
- If the job fails with error, need to collect E2E trace and analyze the related server logs to identify the root cause:
- KBA 1861180 - Collecting an end to end trace in BI Platform 4.x - customer instructions and best practice [Video]
- KBA 2726569 - How to collect an end to end trace with Fiorified Launchpad and BI Platform 4.x
- KBA 2127195 - How to capture an End to End trace of a recurring schedule in BI 4.x
- If issue happens with specific destination, see KBA 3323153 - Destination Troubleshooting in BI 4.x - Collecting logs and reading
For issues when scheduling to File System, check KBA 2477817 and make sure Destination_AllowList.txt has been maintained correctly. Specially in Windows system, make sure the file extension has not been hidden and the file has been created with correct extension and shows correct type, sometimes you are seeing Destination_AllowList.txt but the real file name may be Destination_AllowList.txt.txt, which will not be recognized by the job server.
If issue happens with Allow List content, check jobserverchild log to understand how the target path has been verified. See below log pieces as some checking samples:- destination_disk: file path \\10.10.10.10\peter\sub\DatetimeWebI_58_5976595227465566863.xlsx didn't match C:\peter\
- destination_disk: file path \\10.10.10.10\peter\sub\DatetimeWebI_58_5976595227465566863.xlsx matched \\10.10.10.10\peter\
- destination_disk: file path not found in the whitelist filter: \\10.10.10.10\peter\sub\DatetimeWebI_58_5976595227465566863.xlsx
Audit logs are not inserted in to ADS
- Most of such issues are because the record is already in Auditing DB when BIP tries to insert it to Auditing DB, so the new record can’t be inserted. Check KBA 1733310 for how to resolve this issue.
- CMS medium or higher-level logs will be needed to understand the root cause. To capture the CMS log, You will need to keep only one CMS running if they have multiple CMS running. Set proper log level on only running CMS server and enough keep_num eg: 100 in BO_Trace.ini on the machine of this CMS server, then restart this CMS to force it to load Audit logs. Check the CMS log to see if there is any errors as mentioned in KBA 1733310
Access right issue
- Use CMC -> Query Result feature to query the target report(/folder) to understand the security result and see if needed access rights show granted result.
- Use BI Platform Support Tool -> Security Analyzer to analyze the security setting.
- Check from CMS log and filter the log with value ‘security::IsAllowed’ See https://help.sap.com/docs/SUPPORT_CONTENT/bobjip/3354092081.html
I will keep the blog to be updated with more troubleshooting scenarios.
