How SAP Business Objects gave our team a sleepless night
Hi All,
I am no expert in Business Objects but encountered an issue on Monday that made me learn a lot. In this short blog post i will cover the error, analysis and resolution for the error we got on Monday Morning and got resolved by Tuesday evening only.
Our BO version 4.1 SP4 Update 4
Error
The error began with an email from our customer that the BO isn’t working. We have been facing issues with tomcat lately so we are used to getting these kind of calls. We took it as any other day and started looking at the logs for tomcat crash.
By the way if anyone of you face issues like below in your BO environment, the fix is documented in SAP note 1906557 – Tomcat crash due to EXCEPTION_ACCESS_VIOLATION in the zip.dll with Java_java_util_zip_ZipEntry_initFields
A file named hs_err_pidxxxxx contains
RE version: Java(TM) SE Runtime Environment (6.0_65-b11) (build 6.1.058)
# Java VM: SAP Java Server VM (6.1.058 24.45-b11 Nov 3 2013 00:29:18 – 61_REL – optU – windows amd64 – 6 – bas2:205339 (mixed mode) compressed oops)
# Problematic frame:
# C [zip.dll+0x4c9b] Java_java_util_zip_ZipEntry_initFields+0x1f8b (sp=0x00000000390bf170) (pc=0x000007fefb524c9b)
Coming back to the Monday Error, there were no traces or this kind of file so we thought of restarting BO once as i have been getting alerts for high memory consumption. Post restart the customer could enter BO CMC and BI Launchpad but when refreshed one of the Webi Reports to fetch data from backend BW, we got an error as “Failed to Connect to the OLAP Source (Error: INF)” and the day began with a High priority ticket in my queue. Too much for a Monday morning 😥 😥
Analysis
Our analysis included a number of notes, SCN links, Wiki links and a very high ticket with SAP by Monday evening. The most important blog post was by Ted Ueda which helped me narrow down the issue.
End-to-End Trace and Identifying Root Cause for BI 4.x Web Intelligence
In the CMC — Under Authentication, we maintain a user to backend SAP BW. as soon as we refreshed the report thrice it was getting locked. The users could not login to BI Launchpad if this user was locked using SAP authentication.
Then the blog post mentioned above came to our rescue. With the step by step instructions to look for error in GLF files, I could see the error Name or Password is incorrect. We changed the password, put the password in BO but it did not help 🙁
Now we were quite dependent on SAP’s engineer as we had tried almost everything. Like trying create a new OLAP connection, check new Webi Report, etc, etc but to no avail. It looked more like a treasure hunt with clues and clues but no actual treasure 😡
Another interesting thing was the error when creating a new Webi report. As soon as we clicked on the the OLAP connection we were presented with the Error
java.util.concurrent.ExecutionException: com.businessobjects.sdk.core.server.CommunicationException$UnexpectedServerException: [[error.openSapBwBrowsingSessionFailed] 0] <<?xml version=”1.0″ encoding=”UTF-8″ standalone=”yes”?><ConnectionString>
Resolution
I will call myself too tired to do a note search with error.openSapBwBrowsingSessionFailed as i found numerous SCN posts with similar failed connection to OLAP and no resolution. We have already wasted 1.5 days to resolve this and we were nowhere.
The SAP engineer who had a session with me helped find SAP note 2062091 – In the “Select a BEx query” window, when creating a Web Intelligence report based on BEx, the message “Error: openSapBwBrowsingSessionFailed” appears.
This particular note talks about
- Log into BI Launchpad with the user that was added to the Administrator’s group.
- Create a new Web Intelligence report in the Applet, selecting BEx as the data source, and selecting the OLAP connection on the left window.
- After a few moments, the error “Error: openSapBwBrowsingSessionFailed” appears.
- Once this occurs, the SAP BW account for the Entitlement System becomes or may become locked in BW system, due to too many incorrect attempts.
- If the password for the SAP account that is entered in the Entitlement System tab within the CMC / Authentication SAP section is updated to not contain a special character, the problem does not appear for some environments.
The same was happening with us and the reason was too simple to be true. But as soon as we setup a password without Special characters it was all back to normal. So if you are also getting an issue like this. Try above SAP note and see if it helps.
Such a simple issue gave me and my team a sleepless night but as they say — all is well that ends well 🙂 🙂
Hope you enjoyed reading the post and special thanks to Ted for helping others with such a nice blog and the SAP Engineer who helped us resolve this issue.
Regards
Aditya Khorana
Aditya - Apologies from SAP on the sleepless night! Great write up though with many interesting takeaways.
Thanks Christopher, its always good to have these issues revolving around. Perfect way to learn.. 😉
Hi Aditya,
KBA number below is also realted to your issue.
2138294 - In BI 4.1, the SAP entitlement account becomes locked out when refreshing BICS reports or working within the
CMC Authentication tab.
This could have also helped in faster resolution.
Regards,
Manoj
Hi Sudhakar,
Thanks for your reply, but we never thought in this line that its a bug
The functionality was working fine till Sunday but stopped working on a Monday Morning. The same password with special characters did not cause any trouble for past 1 year.
Regards
Aditya
Hi Aditya,
We faced similar issue in our environment wherein the BW system account running the BICS connection was showing successful tests while testing connections from IDT however when we were running reports, it was throwing error "unable to connect to OLAP source".
The issue was pretty weird because if we pass the BICS connection through message server it was giving error while refreshing report as stated above however passing same connection via application server was working fine.
We were living with the workaround of passing connection through application server and came to know about KBA 2138294 after SAP gave it to us.(We were also clueless as nothing was changed from BO end and reports suddenly started throwing errors.)
In our case the BW system user account's password was changed and this change has not been registered with APS DSL Bridge service leading to issue.
To resolve this a restart of APS DSL bridge service was needed. Once that is done the issue was resolved.
KBA itself explains it more.
Regards,
Manoj Ambre
Why don't you update the JVM used by your Tomcat or just set the compression for the respective connector to off, to avoid these recurring crashs?
Just update it to the SAP JVM 6 release mentioned in the KBA 1906557 (or higher of course) or update it right away to a Java 7 JVM to benefit from performance improvements.
Best Regards
Moritz
Hi Moritz,
Thanks for your reply. We have upgraded dev environment already and waiting for final confirmation for production. SAP doesnt support upgrade of JVM alone for business objects. So the SBOP needs to be upgraded to a higher version altogether.
Thanks Aditya
Hi Aditya,
You're welcome.
You can just update the JVM of Tomcat and leave the JVM used by SBOP as it is.
See for example
How to update JRE/JDK for Tomcat7 on BI 4.1.x Platform
Best Regards
Moritz
Hi Moritz,
the issue described (SAP Note 1906557) was fixed by SAP in their own SAPJVM. The issue is not fixed in the Oracle JVM. So if you're updating to Oracle JVM 7 you will sooner or later face the same issue, but with a slightly different error message.
We're now back to SAPJVM 6 and waiting for BI 4.2 when SAP finally support their own SAPJVM 8...
BR,
Michael
Hi Michael,
The blog post was just an example for the procedure itself.
Since Tomcat 7.0 is supported with Java 7 for BI4.1 (see PAM),
we are running the Tomcat on SAPJVM 7.1.028, without any issues so far.
The issue mentioned in 1906557 is also fixed in this SAPJVM version,
which was verified via SAP incident.
Best Regards
Moritz
Hi Moritz,
yes i know we had a discussion with SAP about the issue and also about the different JVMs..
Tomcat 7 with Oracle Java 7 is supported per PAM
Tomcat 7 with SAPJVM 7 not per PAM. Check the comment in the PAM *6 (only SAPJVM 6.1)..
Sounds crazy? Yes it is..
BR,
Michael
Hi Michael,
*6 doesn't say "only SAPJVM 6.1", it does say "also SAPJVM 6.1". 🙂
I had the same discussion with them, but since PAM just states "Java 7" for Tomcat 7.0 (see page 8), all Java 7 JVMs are supported in my opinion.
Regardless if it's Oracle, IBM, SAP or whatever.
Otherwise they have to have a better explanation here.
There are already 9 exceptions/explanations listed in the footer of this page,
why don't add a 10th?
I told them back then I will handle it as stated until they explicitly list something like "Oracle JDK 7" here. 😛
Anyways, I can just tell that it's technically working perfectly fine in our deployment and we are happy to benefit from the Java 7 improvements, at least for the Tomcat.
And as stated before: it was confirmed by SAP that the mentioned SAPJVM version (and even some earlier versions), doesn't have the compression issue mentioned in 1906557 anymore, which was the core of my initial statement.
Just use SAPJVM >= 6.1.066 or deactivate compression for the connector, if you want to stay on the default SAPJVM 6, instead of watching it repeatedly crashing and doing nothing about it.
Best Regards
Moritz