Skip to Content
Author's profile photo Toby Johnston

How to generate and consume an E2E trace with BI4.x (for non-SolMan landscapes)

For the most up to date instructions for creating an E2E Trace, please refer to:  https://service.sap.com/sap/support/notes/1861180

If you have ever worked on an SAP support message you have without a doubt had to collect BusinessObjects trace logs.  Often you are faced with several challenges and requirements during this process including:

  • You must go into the Central Management Console and manually enable tracing
  • You need to know exactly which components are involved in the workflow you are tracing
  • Traces are huge and contain far more information than is really needed
  • Merging the traces from various components into an ordered and sequenced list is very time consuming
  • Filtering the traces based on thread id, user id, etc then identifying a particular workflow can be downright difficult

Wouldn’t it be great if you could have just one log file that contains only the trace entries related to an end user workflow?  Wouldn’t it be nice if this one log file contained traces from all of the involved servers and web applications in sequential order?  Better yet, what if this could be accomplished without even manually turning on tracing for all of these components?  I know this seems like wishful thinking but with SAP BusinessObjects Business Intelligence Platform 4.x this is now all possible.  The secret sauce that makes this all possible is a new feature implemented in BI 4.x called SAP Passport.

/wp-content/uploads/2016/08/sappassport_153987.png

The SAP Passport is a unique identifier that is generated by a small client application named the SAP Client Plug-In.  Using the SAP Client Plug-in you launch Internet Explorer and during subsequent transactions, the SAP Passport identifier (technically known as the correlation id) is injected into the HTTP header of each request.  The SAP Passport id is then forwarded by the application server to all servers involved in the user’s workflow.  Inside of the SAP Client Plug-in you have the ability to override the default trace level meaning you do not need to manually enable tracing on your BI servers/web applications.  The trace override is passed to each individual server and instructs the server to trace at the specified level for this user’s workflow.  Each trace entry created by the server for the user’s workflow can now be identified by the SAP Passport id thereby enabling a true end to end trace across the BI landscape.

The SAP Client Plug-in is only part of the end to end tracing equation.  The second half of the equation is to bring all of these traces into one continuous view based on the user’s SAP Passport id.  There are several different programs on the internet that could allow you to do this, however we have a new log file reader specifically designed to consume and parse BI 4.x GLF files.  This handy little Java application is aptly named the GLF Viewer and gives you the ability to select multiple GLF files (or a folder), parse these files for a particular string (in this case the SAP Passport id), and build a new log file containing only the information that you want to analyze.

Without further adue, let’s get into the gnitty gritty of how to generate an end to end trace in your BI 4.x landscape.

   

Generate the E2E Trace

1.  Download SAP Client Plug-in 71 SP05 Patch 3 from Note 1435190.  This version is for Internet Explorer 8 and 9 and works only with IE 32-bit.

2.  Copy the SAP Client Plug-in archive to the client where you will be executing the workflow and unzip the files into an empty folder.

3.  Close any open browsers then start the SAP Client Plug-in by executing the file plugin-starter-gui.exe.

4.  Set the “Application” option to Microsoft Internet Explorer and click Launch.

/wp-content/uploads/2016/08/1_154032.png

5.  Internet Explorer (32-bit) will launch and now you must define a few properties before you begin executing your workflow.  Under the “Business Transaction Name” property choose a name that describes the workflow that you are capturing and set the “Next Step TraceLevel” to High.  The “Next Step TraceLevel” setting overrides the current trace level for each component involved in the user workflow.

/wp-content/uploads/2016/08/2_154033.png

6.  Before you start your end to end trace, you should first queue up the browser to the beginning of the problematic workflow that you wish to trace.  In this example, we’ll be tracing a view on-demand request for a Web Intelligence document based on a SQL Server data source (UNX).

7.  When you are ready to begin your workflow, click the “Start Transaction” button.

/wp-content/uploads/2016/08/3_154040.png

8.  After you have completed your workflow press the “Stop Transaction” button.  You can ignore the message “Settings are not valid” as it relates to the SMD Agent settings for Solution Manager.  Click OK to close the SAP Client Plug-in window.

/wp-content/uploads/2016/08/4_154041.png

  

Consume the E2E Trace


For the most up to date E2E trace log reader, please refer to:  http://wiki.scn.sap.com/wiki/display/BOBJ/Flexible+Log+Reader


1.  In the same directory where you executed the SAP Client Plug-in, browse to the /log directory.  Inside of the log directory will be a folder that is named after the value you specified as “Business Transaction Name”.  Browse to the folder that was generated from your tracing session.

/wp-content/uploads/2016/08/5_154042.png

2.  Open the file BusinessTransaction.xml using IE or notepad and search for the string “BusinessTransaction id=”.  Locate the id associated with the BusinessTransaction as this is the unique identifier associated with your E2E trace.

/wp-content/uploads/2016/08/6_154049.png

3.  Since the location of the BILaunchpad or OpenDocument traces is not in the same location as the default logging, you should copy these traces to the default logging directory.  On your application server host, browse to the C:\ drive (or $HOME for Unix) and according to the modified date timestamp, locate the newest folder named SBOPWebapp_BIlaunchpad_*.  Browse into this folder and copy *.glf to the default logging location (for example: C:\Program Files (x86)\SAP BusinessObjects\SAP BusinessObjects Enterprise XI 4.0\logging).

If you have a multi-node landscape, you should copy *.glf from each application server and each BI 4.x node to a centralized folder location.  (note that you may need to set trace level to none or stop the SIA in order to copy the logs from the /logging folder)

4.  Download the GLF Viewer from here (requires an S-USER logon)  This GLF Viewer is from a BI 4.1 build and contains the functionality needed to complete the log parsing.  Older GLF Viewers may not work as shown in this article.

5.  Start the GLF Viewer by executing the file runGLFViewer.bat.

/wp-content/uploads/2016/08/7_154050.png

6.  Inside the GLF Viewer, click File, Open.  Choose Add Files (the Add Folder option will crash the GLFViewer at the time of this writing) and add all *.glf files from the /logging folder (or from your central log folder) that were generated with today’s date.

/wp-content/uploads/2016/08/8_154051.png

7.  Next, confirm that the option “Should merge all into a single tab” is checked.  This option is required to create the end to end view of our traced workflow.

8.  Check the option to “Filter and only read matching entries:” then under the Column option select the field named “DSRRootContextID”, the operator should be “contains”, and in the text box below paste in the BusinessTransaction id that you found previously in step 2, then click OK.

/wp-content/uploads/2016/08/9_154055.png

9.  You are now viewing a continuous end to end trace generated by components in the BI landscape.  These logs have been filtered to show only the transactions generated for your user’s workflow.  Additionally, the trace entries are shown in sequential order as they occur during the workflow.  Analyze the DeviceName column to determine the component that generated each transaction.

/wp-content/uploads/2016/08/10_154056.png

10.  Save your E2E trace as one trace file by clicking File then Export current (filtered) view.

/wp-content/uploads/2016/08/11_154073.png

Assigned tags

      57 Comments
      You must be Logged on to comment or reply to a post.
      Author's profile photo Christopher Vozella
      Christopher Vozella

      Game changer!!!!! Well done Toby and sense a Meet The Expert and hopefully some Tech Ed type pres's in the future on it!!! Bravo!

      Author's profile photo Christopher Vozella
      Christopher Vozella

      The Secret Sauce - that it is!!!

      Author's profile photo Shiva Vishnubatla
      Shiva Vishnubatla

      Thanks for sharing Toby. BI troubleshooting the Real world way.

      Its like a tracer dye used to troubleshoot issues in automative / medical world..

      Author's profile photo Brian Thomas
      Brian Thomas

      Nice blog Toby!  I always wondered how to do this....

      Author's profile photo Denis Konovalov
      Denis Konovalov

      this is actually going to be a great help !!!

      Author's profile photo Scott Broadway
      Scott Broadway

      Great stuff -- thanks!

      Author's profile photo Former Member
      Former Member

      Nice stuff, Really help full.

      Author's profile photo Jonathan Brown
      Jonathan Brown

      Top Notch Toby, thanks for sharing this.  Just testing this out on an issue I am having where I need to track down a performance hit when refreshing a multi-source universe.  So far, it looks to be very helpful in tracking down exactly where the delays are!

      Thanks for sharing. 

      Jb

      Author's profile photo Former Member
      Former Member

      With my 32-bits JVM, I have changed the GLF viewer bat script to use only 1GB of memory:

      -Xmx1g (instead of -Xmx2g)

      Else I had the error:

      "Could not create the Java virtual machine."

      2GB is unfortunately never accepted by 32-bits JVM.

      Hope this helps

      Author's profile photo Former Member
      Former Member

      Thank you for posting this fix. 

      Author's profile photo Chandrakanth Angannagari
      Chandrakanth Angannagari

      Good to see some automation finally for BO troubleshooting.

      I could not understand this line though "note that you may need to set trace level to NONE or stop the SIA in order to copy the log files from the /logging folder)

      Author's profile photo Toby Johnston
      Toby Johnston
      Blog Post Author

      Hi Chandrakanth, thanks for the comment.  Sometimes if the trace file is being actively written to on Windows you may get an error when trying to copy the file to another folder.  By setting the trace level to NONE or stopping the SIA the server will release the lock on the file.

      Cheers

      Toby

      Author's profile photo Toby Johnston
      Toby Johnston
      Blog Post Author

      I have updated the GLF Viewer download to include a bug fix that corrects an error generated when viewing Adaptive Processing Server logs.  Download the new version here

      Author's profile photo Former Member
      Former Member

      Thanks for sharing, Toby. This is awesome...

      Author's profile photo Kevin Geiger
      Kevin Geiger

      Looks very cool.  is there anyway to generate an end-to-end chart, like a pie chart that shows where each chunk of time was spent?

      Author's profile photo Toby Johnston
      Toby Johnston
      Blog Post Author

      Yes, Solution Manager E2E diagnostics does exactly this!

      Author's profile photo Former Member
      Former Member

      Hi Toby,

      Is there anywhere a list of possible entries from the different processing Servers ?

      For example how can i see when a processing server is retrieving the prompts from a db-provider or how can i see when a processing server is retrieving data from the database ?

      is there such a list or can you point me to a location where i can get a list of available messages per service ?

      Best regards and thank you for sharing this very very useful information.

      Manfred

      Author's profile photo Toby Johnston
      Toby Johnston
      Blog Post Author

      Hi Manfred, unfortunately not.  We have internal information such as source code and wiki but there is no externally available information about the specific function calls in tracelog.

      There is a concept to be completed in BI4.1 of trace vs log.  Trace will contain developer information and log will contain information that is understandable by the BI Admin.  So, hopefully it will be easier for the customer in the next version.

      Cheers

      Toby

      Author's profile photo Former Member
      Former Member

      Hi Toby Johnston,

                   Can you please explain me where do i get SBOPWebapp_BIlaunchpad_* files. I have searched for them in the server side but i could able to find. Is there any pre configurations needed for getting them. Please help me.

      Thanks

      Vishnu

      Author's profile photo Toby Johnston
      Toby Johnston
      Blog Post Author

      Hi Vishnu,

      If application server (IE tomcat) is on Windows and runs as local system user then you can find those files on the root of the C:\ drive (of the application server host).   If  it runs as a domain account, check the USER home directory for a .businessobjects folder.

      If the application server is on Unix then these logs will be under $HOME/.businessobjects

      Thanks

      Toby

      Author's profile photo Jonathan Brown
      Jonathan Brown

      Few notes on this. I've used it a LOT now and it really is a game changer.  Has helped me identify many issues.  Thanks again Toby for sharing.

      One common issue we are running into is the keep_num setting in the default bo_trace.ini file.  It is set to 5 by default and that is often not enough if you have a longer running end to end trace.

      For example, I am working with a customer that has a performance issue we are looking at.  E2E tracing is working well, but because the issue spans about 15 minutes, the CMS logs produce about 5 logs every 2 minutes and we keep getting partial logs.  We had to increase the keep_num to 50 or more to get the entire E2E trace.

      This also applies for many of the other servers too.

      So, before you do the end to end trace, you may want to consider doing the following change

      1) make a backup copy of the <install dir>/conf/BO_trace.ini

      2) edit the file and change the keep_num value from 5 to 50 (or more)

      3) save this file.

      4) run the trace.

      this will ensure you get the entire trace from one end to the other. 

      I have also discovered another useful trace that can give you additional tracelog traces.  If you copy the BO_trace.ini file and place it in the default /Tomcat6/ install directory and rename it to:  TraceLog_trace.ini (and then of course modify the file to enable trace at a certain level) then Tomcat will trace all the SAP BOBJ tracelog components to that /tomcat6/ directory.  Quite often we see TraceLog*.glf files already in that tomcat dir but those are just exceptions and fatal errors by default.  This TraceLog_trace.ini allows us to get a more detailed log from Tomcat.

      A lot of these entries are duplicate to the ones we see in the WebApps (BiLaunchpad, CMC, ws) logging dirs, but I have definitely seen more details in these logs.

      Thanks

      Jb

      Author's profile photo Toby Johnston
      Toby Johnston
      Blog Post Author

      Thanks JB!

      You can also use an if statement in your bo_trace.ini if you don't want all your processes to use keep_num=50.  For example:

      if (process == "webiserver_bi40.WebIntelligenceProcessingServer" || process == "cms_bi40.CentralManagementServer")

      {

      keep_num=50

      }

      Author's profile photo Jonathan Brown
      Jonathan Brown

      Great point Toby!  I've been meaning to narrow down the syntax for that for a while now.  Thanks for adding it in here!

      Author's profile photo Toby Johnston
      Toby Johnston
      Blog Post Author

      For BI4 Java processes, they are started with a java property -Dtracelog.process= which in the GLF is actually the "DeviceName". 

      So you can also do:

      if (process == <DEVICE_NAME>)

      or

      if (process == <BI_SERVER_NAME>)

      Author's profile photo Former Member
      Former Member

      Hi Toby,

      Thanks for the Wonderful post.

      I am trying to generate a filtered view of the logs but I am having issues launching the GLF Viewer.

      It fails with the following error:

      "Could not create the java virtual machine"

      I would appreciate if you can help me fix this.

      Thanks,

      Vikram

      Author's profile photo Toby Johnston
      Toby Johnston
      Blog Post Author

      Hi Vikram,

      You need to install JRE 6.  Then create user environment variable JAVA_HOME and set it to the location of your JRE6.  Then double click on the batch file included with the zip.

      Thanks

      Toby

      Author's profile photo Former Member
      Former Member

      Hi Toby,

      Thanks for the reply but that didn't work.

      However good news, I found a way to get this working:

      I copied the GLFviewer.jar to the following directory:

      I:\Program Files (x86)\SAP BusinessObjects\SAP BusinessObjects Enterprise XI 4.0\win64_x64\jdk\jre\bin

      Then navigated to this path in the Command Prompt and executed the following:

      I:\Program Files (x86)\SAP BusinessObjects\SAP BusinessObjects Enterprise XI 4.0

      \win64_x64\jdk\jre\bin>javaw.exe -jar GLFviewer.jar

      That worked.

      Thanks again for the wonderful post and a special thanks for the syntax to modify the bo_trace.ini for tracing certain processes only.

      Cheers,

      Vikram.V

      Author's profile photo Toby Johnston
      Toby Johnston
      Blog Post Author

      Thanks for the kind words...

      About your comment, you need to add the path to your java bin folder to the Path user environment variable.  Then you can run the java command from anywhere on your filesystem (in this case the folder where you unzipped glfviewer)

      For example

      Path=%Path%;%JAVA_HOME%\bin

      Author's profile photo Former Member
      Former Member
      Author's profile photo Former Member
      Former Member

      Hi Toby,

      Thanks for a very interesting article.

      However, it seems the download link for the GLF Viewer has expired.

      Can you please re-upload?

      Thanks!

      Jan Terje

      Author's profile photo Toby Johnston
      Toby Johnston
      Blog Post Author

      Hi Jan Terje,

      I uploaded the file again for you.

      https://sapmats-us.sap-ag.de/download/download.cgi?id=XJGD9FNGM95LQX95SIQLSCZJV5CBVEM2LV1JY32KSIQD9BRO3R

      Don't forget to rate my blog 🙂

      Cheers

      Toby

      Author's profile photo Former Member
      Former Member

      Thanks!

      Author's profile photo Rodrigo Caparroz
      Rodrigo Caparroz

      Nice! Thanks for sharing!

      Author's profile photo Former Member
      Former Member

      Hi Toby, thank you very much for this extremely useful tool !

      I get this error msg while trying to load some files (eg:Webi Processing and Platform Search servers): Wrong number of columns in record, expected 34, but is 27

      BI4 version is SP02 Patch 21

      FYI, I use 32b JVM

      Author's profile photo Denis Konovalov
      Denis Konovalov

      it is a problem that is being investigated. you'll need to manualy view those files that produce this error.

      Author's profile photo Toby Johnston
      Toby Johnston
      Blog Post Author

      Hi Pascal,

      Please see the following SAP note for a solution to the "Wrong Number of columns" error.

      1891488 - Error "Wrong number of columns in
      record" when attempting to open a GLF trace file in GLF Viewer

      https://service.sap.com/sap/support/notes/1891488

      Don't forget to rate this blog if you found it useful!

      Cheers

      Toby

      Author's profile photo Former Member
      Former Member

      Hi Toby,

      Very well done post. I can see exactly how this type of tracing can be very useful in the real world. You've also done a great job of explaining everything into great detail.

      Look forward to reading more from you.

      -Victor

      Author's profile photo Former Member
      Former Member

      Outstanding article Toby.

      Using the tracing techniques as a point of entry for remote debugging looks promising.

      Cheers

      -Mauricio.

      Author's profile photo Toby Johnston
      Toby Johnston
      Blog Post Author

      Hey Mauricio,

      Check out note 1861180 Customer Instructions and best practice for collecting a BI Platform 4.x end to end trace

      It has all the latest details needed to generate and collect a proper E2E trace from BI 4.x. 

      Cheers

      Toby

      Author's profile photo Bob Zenker
      Bob Zenker

      When trying to download this tool my virus scan software says there is a virus associated with the file!  Please check out the file. 😯

      Author's profile photo Brian Thomas
      Brian Thomas

      Hi Bob,

      Which AV do you have and which file?  I've downloaded and scanned both and they'r eok for me.

      Author's profile photo Toby Johnston
      Toby Johnston
      Blog Post Author

      Hi Bob,

      The SAP Client Plug-in intercepts and injects headers into HTTP traffic between the browser and server. Therefore it may be considered as potentially un-safe by virus protection software.  There is no actual virus in the SAP Client Plug-in 😉

      Refer to:

      1879918 - Virus scan detects SAP Client Plug-in as a trojan or
      virus

      Author's profile photo Mihir T
      Mihir T

      Hi Toby,

      We have a critical Crystal Report which is scheduled to run every morning at a certain time, but sometimes these schedules are delayed whereas sometimes they are on time.

      Can we use E2E trace to trace this schedule ? asking because you have mentioned that "SAP Client Plug-in intercepts and injects headers into HTTP traffic ", but in our case we already have the schedules configured and we just require monitoring and there wont be any HTTP interaction.Can this be achieved with E2e ? please let me know.

      Thanks !

      Author's profile photo Toby Johnston
      Toby Johnston
      Blog Post Author

      You could try to reschedule the recurring instance using the SAP Client Plugin.  I've not tested a recurring instance specifically but it's worth a shot.

      More specifically, for your issue I would turn on HIGH trace for all CMS services and all Crystal Report Job Server just before those reports are scheduled to kick off.  Then after the reports have executed (and were delayed) collect the traces.  We need to look at the CMS scheduler thread, and Job Server communication / processing using the instance SI_ID as a filter.  It will provide clue as to why it was delayed.

      Author's profile photo Mihir T
      Mihir T

      Thanks a lot Toby !

      That really helps !

      -

      Mihir

      Author's profile photo Alan Han
      Alan Han

      This is a great tool! Thanks Toby!

      Author's profile photo Dayanand Rao
      Dayanand Rao

      very useful tool. Thanks 🙂

      Author's profile photo Former Member
      Former Member

      Is anyone able to see the screenshots for each step? Only step 4 displays for me; the rest show an error icon.

      Author's profile photo John Clark
      John Clark

      Same for me.  I can only get the image associated with step 4 to show.

      Author's profile photo Former Member
      Former Member

      KBA 1861180 - Customer instructions and best practice for collecting a BI Platform 4.x end to end trace. It has some of the screenshots in there.

      Author's profile photo Toby Johnston
      Toby Johnston
      Blog Post Author

      Hi Pedro,

      Not sure what happened.  When I am back in the office I will fix the images however, you should not follow this article anymore.  For most up to date E2E Trace instructions, please refer to the kb article that Patrick mentioned:

      KBA 1861180 - Customer instructions and best practice for collecting a BI Platform 4.x end to end trace

      http://service.sap.com/sap/support/notes/1861180

      Author's profile photo Former Member
      Former Member

      Excellent, thanks for clarifying.

      Author's profile photo Michael Neville
      Michael Neville

      KBA 1861180 also contains a link to download an updated version of this tool that is compatible with Internet Explorer 11.

      Author's profile photo Former Member
      Former Member

      Exellent article, Toby. You mentioned "the SAP Passport identifier (technically known as the correlation id) is injected into the HTTP header of each request". We are building a custom portal using the RESTful API and would like to inject this header ourselves. Do you happen to know the exact name/syntax of the HTTP header? Thanks!

      Author's profile photo Toby Johnston
      Toby Johnston
      Blog Post Author

      Hi Don,

      Check out this SAP Note for generating passport via SDK

      http://service.sap.com/sap/support/notes/1608231

      Also, please refer to this kb article for most up to date instructions on E2E tracing with BI as this article is a bit out dated.

      http://service.sap.com/sap/support/notes/1861180

      Thanks

      Toby

      Author's profile photo Former Member
      Former Member

      Hello,

      Thanks for the nice document.

      I am unable to export ( export current filtered view)  using GLFviewe. The GLFviewer gets closed. Please help.

      regards

      Kiran

      Author's profile photo Toby Johnston
      Toby Johnston
      Blog Post Author

      You might try to increase the amount of Heap that GLFViewer can use to 3gb.  Edit the .bat file and add -Xmx3g

      start "" /D. "%JVM_EXE%" -Xmx3g -jar "%~0\..\GLFviewer.jar"