Skip to Content
For the most up to date instructions for creating an E2E Trace, please refer to:  https://service.sap.com/sap/support/notes/1861180

If you have ever worked on an SAP support message you have without a doubt had to collect BusinessObjects trace logs.  Often you are faced with several challenges and requirements during this process including:

  • You must go into the Central Management Console and manually enable tracing
  • You need to know exactly which components are involved in the workflow you are tracing
  • Traces are huge and contain far more information than is really needed
  • Merging the traces from various components into an ordered and sequenced list is very time consuming
  • Filtering the traces based on thread id, user id, etc then identifying a particular workflow can be downright difficult

Wouldn’t it be great if you could have just one log file that contains only the trace entries related to an end user workflow?  Wouldn’t it be nice if this one log file contained traces from all of the involved servers and web applications in sequential order?  Better yet, what if this could be accomplished without even manually turning on tracing for all of these components?  I know this seems like wishful thinking but with SAP BusinessObjects Business Intelligence Platform 4.x this is now all possible.  The secret sauce that makes this all possible is a new feature implemented in BI 4.x called SAP Passport.

/wp-content/uploads/2016/08/sappassport_153987.png

The SAP Passport is a unique identifier that is generated by a small client application named the SAP Client Plug-In.  Using the SAP Client Plug-in you launch Internet Explorer and during subsequent transactions, the SAP Passport identifier (technically known as the correlation id) is injected into the HTTP header of each request.  The SAP Passport id is then forwarded by the application server to all servers involved in the user’s workflow.  Inside of the SAP Client Plug-in you have the ability to override the default trace level meaning you do not need to manually enable tracing on your BI servers/web applications.  The trace override is passed to each individual server and instructs the server to trace at the specified level for this user’s workflow.  Each trace entry created by the server for the user’s workflow can now be identified by the SAP Passport id thereby enabling a true end to end trace across the BI landscape.

The SAP Client Plug-in is only part of the end to end tracing equation.  The second half of the equation is to bring all of these traces into one continuous view based on the user’s SAP Passport id.  There are several different programs on the internet that could allow you to do this, however we have a new log file reader specifically designed to consume and parse BI 4.x GLF files.  This handy little Java application is aptly named the GLF Viewer and gives you the ability to select multiple GLF files (or a folder), parse these files for a particular string (in this case the SAP Passport id), and build a new log file containing only the information that you want to analyze.

Without further adue, let’s get into the gnitty gritty of how to generate an end to end trace in your BI 4.x landscape.

   

Generate the E2E Trace

1.  Download SAP Client Plug-in 71 SP05 Patch 3 from Note 1435190.  This version is for Internet Explorer 8 and 9 and works only with IE 32-bit.

2.  Copy the SAP Client Plug-in archive to the client where you will be executing the workflow and unzip the files into an empty folder.

3.  Close any open browsers then start the SAP Client Plug-in by executing the file plugin-starter-gui.exe.

4.  Set the “Application” option to Microsoft Internet Explorer and click Launch.

/wp-content/uploads/2016/08/1_154032.png

5.  Internet Explorer (32-bit) will launch and now you must define a few properties before you begin executing your workflow.  Under the “Business Transaction Name” property choose a name that describes the workflow that you are capturing and set the “Next Step TraceLevel” to High.  The “Next Step TraceLevel” setting overrides the current trace level for each component involved in the user workflow.

/wp-content/uploads/2016/08/2_154033.png

6.  Before you start your end to end trace, you should first queue up the browser to the beginning of the problematic workflow that you wish to trace.  In this example, we’ll be tracing a view on-demand request for a Web Intelligence document based on a SQL Server data source (UNX).

7.  When you are ready to begin your workflow, click the “Start Transaction” button.

/wp-content/uploads/2016/08/3_154040.png

8.  After you have completed your workflow press the “Stop Transaction” button.  You can ignore the message “Settings are not valid” as it relates to the SMD Agent settings for Solution Manager.  Click OK to close the SAP Client Plug-in window.

/wp-content/uploads/2016/08/4_154041.png

  

Consume the E2E Trace


For the most up to date E2E trace log reader, please refer to:  http://wiki.scn.sap.com/wiki/display/BOBJ/Flexible+Log+Reader


1.  In the same directory where you executed the SAP Client Plug-in, browse to the /log directory.  Inside of the log directory will be a folder that is named after the value you specified as “Business Transaction Name”.  Browse to the folder that was generated from your tracing session.

/wp-content/uploads/2016/08/5_154042.png

2.  Open the file BusinessTransaction.xml using IE or notepad and search for the string “BusinessTransaction id=”.  Locate the id associated with the BusinessTransaction as this is the unique identifier associated with your E2E trace.

/wp-content/uploads/2016/08/6_154049.png

3.  Since the location of the BILaunchpad or OpenDocument traces is not in the same location as the default logging, you should copy these traces to the default logging directory.  On your application server host, browse to the C:\ drive (or $HOME for Unix) and according to the modified date timestamp, locate the newest folder named SBOPWebapp_BIlaunchpad_*.  Browse into this folder and copy *.glf to the default logging location (for example: C:\Program Files (x86)\SAP BusinessObjects\SAP BusinessObjects Enterprise XI 4.0\logging).

If you have a multi-node landscape, you should copy *.glf from each application server and each BI 4.x node to a centralized folder location.  (note that you may need to set trace level to none or stop the SIA in order to copy the logs from the /logging folder)

4.  Download the GLF Viewer from here (requires an S-USER logon)  This GLF Viewer is from a BI 4.1 build and contains the functionality needed to complete the log parsing.  Older GLF Viewers may not work as shown in this article.

5.  Start the GLF Viewer by executing the file runGLFViewer.bat.

/wp-content/uploads/2016/08/7_154050.png

6.  Inside the GLF Viewer, click File, Open.  Choose Add Files (the Add Folder option will crash the GLFViewer at the time of this writing) and add all *.glf files from the /logging folder (or from your central log folder) that were generated with today’s date.

/wp-content/uploads/2016/08/8_154051.png

7.  Next, confirm that the option “Should merge all into a single tab” is checked.  This option is required to create the end to end view of our traced workflow.

8.  Check the option to “Filter and only read matching entries:” then under the Column option select the field named “DSRRootContextID”, the operator should be “contains”, and in the text box below paste in the BusinessTransaction id that you found previously in step 2, then click OK.

/wp-content/uploads/2016/08/9_154055.png

9.  You are now viewing a continuous end to end trace generated by components in the BI landscape.  These logs have been filtered to show only the transactions generated for your user’s workflow.  Additionally, the trace entries are shown in sequential order as they occur during the workflow.  Analyze the DeviceName column to determine the component that generated each transaction.

/wp-content/uploads/2016/08/10_154056.png

10.  Save your E2E trace as one trace file by clicking File then Export current (filtered) view.

/wp-content/uploads/2016/08/11_154073.png

To report this post you need to login first.

57 Comments

You must be Logged on to comment or reply to a post.

  1. Jonathan Brown

    Top Notch Toby, thanks for sharing this.  Just testing this out on an issue I am having where I need to track down a performance hit when refreshing a multi-source universe.  So far, it looks to be very helpful in tracking down exactly where the delays are!

    Thanks for sharing. 

    Jb

    (0) 
  2. JeanPierre Matsumoto

    With my 32-bits JVM, I have changed the GLF viewer bat script to use only 1GB of memory:

    -Xmx1g (instead of -Xmx2g)

    Else I had the error:

    “Could not create the Java virtual machine.”

    2GB is unfortunately never accepted by 32-bits JVM.

    Hope this helps

    (0) 
  3. Chandrakanth Angannagari

    Good to see some automation finally for BO troubleshooting.

    I could not understand this line though “note that you may need to set trace level to NONE or stop the SIA in order to copy the log files from the /logging folder)

    (0) 
    1. Toby Johnston Post author

      Hi Chandrakanth, thanks for the comment.  Sometimes if the trace file is being actively written to on Windows you may get an error when trying to copy the file to another folder.  By setting the trace level to NONE or stopping the SIA the server will release the lock on the file.

      Cheers

      Toby

      (0) 
  4. Toby Johnston Post author

    I have updated the GLF Viewer download to include a bug fix that corrects an error generated when viewing Adaptive Processing Server logs.  Download the new version here

    (0) 
  5. Manfred Schwarz

    Hi Toby,

    Is there anywhere a list of possible entries from the different processing Servers ?

    For example how can i see when a processing server is retrieving the prompts from a db-provider or how can i see when a processing server is retrieving data from the database ?

    is there such a list or can you point me to a location where i can get a list of available messages per service ?

    Best regards and thank you for sharing this very very useful information.

    Manfred

    (0) 
    1. Toby Johnston Post author

      Hi Manfred, unfortunately not.  We have internal information such as source code and wiki but there is no externally available information about the specific function calls in tracelog.

      There is a concept to be completed in BI4.1 of trace vs log.  Trace will contain developer information and log will contain information that is understandable by the BI Admin.  So, hopefully it will be easier for the customer in the next version.

      Cheers

      Toby

      (0) 
  6. Vishnuvardhan Reddy Tirumala Reddy

    Hi Toby Johnston,

                 Can you please explain me where do i get SBOPWebapp_BIlaunchpad_* files. I have searched for them in the server side but i could able to find. Is there any pre configurations needed for getting them. Please help me.

    Thanks

    Vishnu

    (0) 
    1. Toby Johnston Post author

      Hi Vishnu,

      If application server (IE tomcat) is on Windows and runs as local system user then you can find those files on the root of the C:\ drive (of the application server host).   If  it runs as a domain account, check the USER home directory for a .businessobjects folder.

      If the application server is on Unix then these logs will be under $HOME/.businessobjects

      Thanks

      Toby

      (0) 
  7. Jonathan Brown

    Few notes on this. I’ve used it a LOT now and it really is a game changer.  Has helped me identify many issues.  Thanks again Toby for sharing.

    One common issue we are running into is the keep_num setting in the default bo_trace.ini file.  It is set to 5 by default and that is often not enough if you have a longer running end to end trace.

    For example, I am working with a customer that has a performance issue we are looking at.  E2E tracing is working well, but because the issue spans about 15 minutes, the CMS logs produce about 5 logs every 2 minutes and we keep getting partial logs.  We had to increase the keep_num to 50 or more to get the entire E2E trace.

    This also applies for many of the other servers too.

    So, before you do the end to end trace, you may want to consider doing the following change

    1) make a backup copy of the <install dir>/conf/BO_trace.ini

    2) edit the file and change the keep_num value from 5 to 50 (or more)

    3) save this file.

    4) run the trace.

    this will ensure you get the entire trace from one end to the other. 

    I have also discovered another useful trace that can give you additional tracelog traces.  If you copy the BO_trace.ini file and place it in the default /Tomcat6/ install directory and rename it to:  TraceLog_trace.ini (and then of course modify the file to enable trace at a certain level) then Tomcat will trace all the SAP BOBJ tracelog components to that /tomcat6/ directory.  Quite often we see TraceLog*.glf files already in that tomcat dir but those are just exceptions and fatal errors by default.  This TraceLog_trace.ini allows us to get a more detailed log from Tomcat.

    A lot of these entries are duplicate to the ones we see in the WebApps (BiLaunchpad, CMC, ws) logging dirs, but I have definitely seen more details in these logs.

    Thanks

    Jb

    (0) 
    1. Toby Johnston Post author

      Thanks JB!

      You can also use an if statement in your bo_trace.ini if you don’t want all your processes to use keep_num=50.  For example:

      if (process == “webiserver_bi40.WebIntelligenceProcessingServer” || process == “cms_bi40.CentralManagementServer”)

      {

      keep_num=50

      }

      (0) 
        1. Toby Johnston Post author

          For BI4 Java processes, they are started with a java property -Dtracelog.process= which in the GLF is actually the “DeviceName”. 

          So you can also do:

          if (process == <DEVICE_NAME>)

          or

          if (process == <BI_SERVER_NAME>)

          (0) 
  8. Vikram Veeranagari

    Hi Toby,

    Thanks for the Wonderful post.

    I am trying to generate a filtered view of the logs but I am having issues launching the GLF Viewer.

    It fails with the following error:

    “Could not create the java virtual machine”

    I would appreciate if you can help me fix this.

    Thanks,

    Vikram

    (0) 
    1. Toby Johnston Post author

      Hi Vikram,

      You need to install JRE 6.  Then create user environment variable JAVA_HOME and set it to the location of your JRE6.  Then double click on the batch file included with the zip.

      Thanks

      Toby

      (0) 
      1. Vikram Veeranagari

        Hi Toby,

        Thanks for the reply but that didn’t work.

        However good news, I found a way to get this working:

        I copied the GLFviewer.jar to the following directory:

        I:\Program Files (x86)\SAP BusinessObjects\SAP BusinessObjects Enterprise XI 4.0\win64_x64\jdk\jre\bin

        Then navigated to this path in the Command Prompt and executed the following:

        I:\Program Files (x86)\SAP BusinessObjects\SAP BusinessObjects Enterprise XI 4.0

        \win64_x64\jdk\jre\bin>javaw.exe -jar GLFviewer.jar

        That worked.

        Thanks again for the wonderful post and a special thanks for the syntax to modify the bo_trace.ini for tracing certain processes only.

        Cheers,

        Vikram.V

        (0) 
        1. Toby Johnston Post author

          Thanks for the kind words…

          About your comment, you need to add the path to your java bin folder to the Path user environment variable.  Then you can run the java command from anywhere on your filesystem (in this case the folder where you unzipped glfviewer)

          For example

          Path=%Path%;%JAVA_HOME%\bin

          (0) 
  9. Jan Terje Hansen

    Hi Toby,

    Thanks for a very interesting article.

    However, it seems the download link for the GLF Viewer has expired.

    Can you please re-upload?

    Thanks!

    Jan Terje

    (0) 
  10. Pascal GOURLAOUEN

    Hi Toby, thank you very much for this extremely useful tool !

    I get this error msg while trying to load some files (eg:Webi Processing and Platform Search servers): Wrong number of columns in record, expected 34, but is 27

    BI4 version is SP02 Patch 21

    FYI, I use 32b JVM

    (0) 
  11. Victor Munker

    Hi Toby,

    Very well done post. I can see exactly how this type of tracing can be very useful in the real world. You’ve also done a great job of explaining everything into great detail.

    Look forward to reading more from you.

    -Victor

    (0) 
  12. Bob Zenker

    When trying to download this tool my virus scan software says there is a virus associated with the file!  Please check out the file. 😯

    (0) 
      1. Mihir T

        Hi Toby,

        We have a critical Crystal Report which is scheduled to run every morning at a certain time, but sometimes these schedules are delayed whereas sometimes they are on time.

        Can we use E2E trace to trace this schedule ? asking because you have mentioned that “SAP Client Plug-in intercepts and injects headers into HTTP traffic “, but in our case we already have the schedules configured and we just require monitoring and there wont be any HTTP interaction.Can this be achieved with E2e ? please let me know.

        Thanks !

        (0) 
        1. Toby Johnston Post author

          You could try to reschedule the recurring instance using the SAP Client Plugin.  I’ve not tested a recurring instance specifically but it’s worth a shot.

          More specifically, for your issue I would turn on HIGH trace for all CMS services and all Crystal Report Job Server just before those reports are scheduled to kick off.  Then after the reports have executed (and were delayed) collect the traces.  We need to look at the CMS scheduler thread, and Job Server communication / processing using the instance SI_ID as a filter.  It will provide clue as to why it was delayed.

          (0) 
    1. Toby Johnston Post author

      Hi Pedro,

      Not sure what happened.  When I am back in the office I will fix the images however, you should not follow this article anymore.  For most up to date E2E Trace instructions, please refer to the kb article that Patrick mentioned:

      KBA 1861180 – Customer instructions and best practice for collecting a BI Platform 4.x end to end trace

      http://service.sap.com/sap/support/notes/1861180

      (0) 
  13. Don Collins

    Exellent article, Toby. You mentioned “the SAP Passport identifier (technically known as the correlation id) is injected into the HTTP header of each request”. We are building a custom portal using the RESTful API and would like to inject this header ourselves. Do you happen to know the exact name/syntax of the HTTP header? Thanks!

    (0) 
  14. KIRAN CHAUDHARI

    Hello,

    Thanks for the nice document.

    I am unable to export ( export current filtered view)  using GLFviewe. The GLFviewer gets closed. Please help.

    regards

    Kiran

    (0) 
    1. Toby Johnston Post author

      You might try to increase the amount of Heap that GLFViewer can use to 3gb.  Edit the .bat file and add -Xmx3g

      start “” /D. “%JVM_EXE%” -Xmx3g -jar “%~0\..\GLFviewer.jar”

      (0) 

Leave a Reply