Recently a support ticket was raised for a "portal runtime error"; the issue occured for a number of end-users, but the Basis support team was unable to reproduce the issue. Basis support team was unable to reproduce because the iView contents is generated dynamically based on access roles, so the Basis support team does not have access to the object generating the error.
--
! Portal runtime error.
An exception occured while processing your request. Send the exception Id to your portal administrator.
Exception ID: 08:39_13/07/11_0028_115502652
Refer to the log file for details about this exception.
--
Basis support team started analysis of the defaultTrace log-files, which is a significant task because in total there are over 4 GB of log-files (22 server nodes x 20 log files x 10 MB). This portal runs on AIX UNIX and we have some search scripts to help with these tasks, or alternatively Visual Administrator / Log Viewer or NWA can be used. In this case we requested the Portal support team to reproduce the issue and requested node-information, this points to the server node narrowing down the defaultTrace files to look at. In the defaultTrace files the error "SQLException is not transient" occured and confirmed that the USERID and timestamp related to when the issue occured;
--
#1.5^H#00145E76AB02008000000027000930180004A7EF5AF178A1#1310546371638#com.sap.portal.prt.request#sap.com/irj#com.sap.portal.prt.request#USERID#144109##41495487A80B11E
0B8D4000006E26E3C#41495487a80b11e0b8d4000006e26e3c-0#41495487a80b11e0b8d4000006e26e3c#Thread[PRT-Async 2,5,PRT-Async]##0#0#Error#1#/System/Server#Java###Exception in PortalRequestManager.dispatchRequest without timeout
[EXCEPTION]{0}#1#PcdRuntimeException: SQLException is not transient, error code = -803, sql state = 23505
DB2 SQL Error: SQLCODE=-803, SQLSTATE=23505, SQLERRMC=3;SAPEPXDB.EP_OBJECTS, DRIVER=3.53.95
com.sapportals.portal.pcd.pl.PlRuntimeException: SQLException is not transient, error code = -803, sql state = 23505
DB2 SQL Error: SQLCODE=-803, SQLSTATE=23505, SQLERRMC=3;SAPEPXDB.EP_OBJECTS, DRIVER=3.53.95
at com.sapportals.portal.pcd.pl.PlFactory.handleUnexpectedSQLError(PlFactory.java:1472)
at com.sapportals.portal.pcd.pl.PlFactory.checkDeadlockResolution(PlFactory.java:1484)
at com.sapportals.portal.pcd.gl.xfs.XfsContext.createChildAndNotifyAtomicName(XfsContext.java:1065)
at com.sapportals.portal.pcd.gl.xfs.XfsContext.createChildAtomicName(XfsContext.java:1007)
at com.sapportals.portal.pcd.gl.xfs.XfsContext.createChild(XfsContext.java:970)
at com.sapportals.portal.pcd.gl.xfs.XfsContext.createSubcontext(XfsContext.java:842)
--
SAP Note 1352628 - Content is not accessible, after content change or import shows a similar error, but for an Oracle system. This portal runs on IBM DB2 so the error message gets displayed slightly differently, IBM documentation confirms that SQLSTATE=23505 means "A violation of the constraint imposed by a unique index or a unique constraint occured.".
The workaround suggested in SAP Note 1352628 is to clear the PCD cache; because we didn't know exactly which object was causing the issue the whole PCD cache was cleared (investigation into the exact PCD object causing the issue would take time, and we were aiming to restore service asap). Clearing the complete PCD cache occurs for the current node only, and was first done on the node where the Portal support team was reproducting the issue. After clearing the cache, the issue did no longer occur, so the PCD cache was cleared on the remaining nodes as well.
NB: clearning the PCD cache will have a performance impact as the objects are re-read from the database on first access. Basis and Portal support teams helped mitigating against this performance impact by manually connecting to the server nodes after the PCD cache was cleared.
A customer message was raised with SAP to investigate the root cause. SAP advised that the issue described in SAP Note 1352628 is different, but that the symptoms are the same and the workaround still applies. SAP have advised that the issue may occur if there are 2 pages that are delta-linked and have an iView with the same name, or if a workset is transported with a delta-link to a role which was not included in the transport. Either scenario is rare, and SAP are working on a more permanent fix to be included in a future support package (probably NW702 SP11).
Lessons learned:
1) Perform analysis of log-files, the initial error message was very generic and the "real" error was only found after log-file analysis;
2) Search for SAP notes, even though the error was slightly different, the symptoms were the same and the workaround still applied;
3) Involve SAP, especially if further analysis is required (even after the incident was fixed).
