Skip to Content
Author's profile photo Jim Spath

End to End Root Cause Analysis – I’m Virtually Trainable (Day 0.6)

Day 1 2 THREE 4 5


Class notes


Now that the nine of us have been together for 2 entire business days, a new morning is like a reunion, rather than a random bus trip across town.  I shared the link to the blog post on yesterday’s class, which the instructor said was a great write-up. He said he was cautious at first, like he was a chef finding out that a newspaper restaurant critic was visiting and was going to review his creations.  But after seeing more of my off-the-cuff style, he said he thought this was a great feedback mechanism – no need to wait until the end-of-class review survey.


Then I got a Linked-In invite from a classmate (whom I’ve obscured below), which I’ve accepted.  The instructor was thinking of ways we could collaborate using other SCN tools also.





I like that Vijay Vijayasankar is my link to the classmate, and curious who their mutual connections are.  I can’t see, based on current security.


The next class project, after a quick review of prior material, was a diversion into the management console, launched via web address.  Here we learned that one of the two demo systems was working fine.  




On the other hand, the second instance was not well.  Yellow lights were lit, and after a bit of navigating, we could see processes were down.





The next screen shot is from the instructors session, through the webcast (the above are from my remote desktop session.   You can see the trace file with the somewhat hilarious yet at the same time ominous phrase “I better EXIT before I do any (more) damage … “. [ side bar – link to an SCN forum post with this phrase, from 2007: Sol Man  4.0 installation – dispatcher stops ]



 Thread Dump Analysis


Finally (for some of you), on to looking at Java thread dumps.  This is more on the how, but still some of the why.  The when is another story.



image =

I’ll fast forward here, from the above menu starting point, to one of the more interesting views in the interface.  There is an “active” filter, created by typing “waiting for monitor” in the blank area beneath the top of the Thread State column, that displays just those 2 rows.  I had trouble with this, expecting that typing partial words would also work, but didn’t seem to get the same view.


I selected one of the rows for illustration.  The class joke was “why don’t developers name their bad logic sections with names like SimpleDeadlock?”




This is the same area, but a different tab.  In this view, there are nice, bright, red lights showing where problems are likely to be.


The columns represent different dump periods, which might be triggered manually a few seconds or minutes apart in order to zero in on the bugs.  At least, that’s my theory.




Drilling in a little farther, here are “thread details” for one of the bad setups seen above.  This red light shows the locked state of the thread.  Very easy, with a little help from my friends.




End to End Trace Analysis


This procedure requires a download of the “Internet Explorer Plug In”.  The class notes mention 2 SAP Notes; I found a third that may be newer.

The last note was not current in English when I looked at it, but I’m confident it will be updated shortly.  What I downloaded:

$ c:/bin/unzip -v  c:/.../SAP*zip
Archive:  c:/.../
 Length   Method    Size  Ratio   Date   Time   CRC-32    Name
--------  ------  ------- -----   ----   ----   ------    ----
 1313867  Defl:N  1264587   4%  08-20-10 07:48  d644e9f0  SAPClientPlugin.exe


Running this unpacked a set of files, along with other files created when running it for testing.

     688 2009-11-09 17:53 ie-https_sample.cmd
 1313867 2010-08-20 07:48 SAPClientPlugin.exe
 933888 2010-08-20 13:48 http-proxy.dll
  864256 2010-08-20 13:48 rfc-proxy.dll
 1019904 2010-08-20 13:48 plugin-framework.dll
  798720 2010-08-20 13:48 plugin-starter.exe
 1916928 2010-08-20 13:48 plugin-starter-gui.exe
      86 2010-09-22 15:30
       0 2010-09-22 15:31 log


The “log” directory gets the BusinessTransaction.xml file, which will upload to Solution Manager, assuming the SMD host, port, and infrastructure are working right (mine are not).




Trace collection


Once the traces are collected, they need to be transferred into the analysis tool.  I haven’t done this yet, so the next few shots are from the instructor’s screen.


First, the upload or transfer phase “in progress since 44 seconds [sic]”.




Then, a easy to view display of the trace summary, with drilldown sections available.  This part I displayed, based on the preloaded trace data.





The Server Analysis shows timings on the backend, similar to ST05, but for Java in this case.  I’ve superimposed several screen sections into one in order to point out several mouse-over-pop-up icons.





Then, finally, it’s back to the “Tree View” in the Wily Introscope Workstation.  It’s getting late, and I’ve lost the bread crumb trail to get here.  This was launched from the Wily Introscope console as a “New Transaction Trace Session.”  We looked for anything over 5 seconds.





What’s missing?

  • “Classic” ABAP workload and trace analysis.  We talked about it, but I am not sharing any screen shots.  They were all from the SAP GUI.
  • BMC AppSight.  We talked about it, but did not see or run any demos.  It doesn’t seem to be installed in the demo remote desktop, meaning more setup is required to even think about it.
  • Network analysis.  This is not covered in the class.  I promised to share links to blogs I wrote recently on my network investigations – see below.




 SAP_VLC_E2E100 is the Twitter hash tag I used for a few stray thoughts during class.


I did most of the screen shots locally, or via the Citrix desktop, not the web cast view, so they would be clearer.  I did not  capture anything like a slide view, as that is drifting into copyright land.  I don’t think my demos are worth much, except as a reminder to myself how I got there.  The instructor was starting to hint “I’d like to see this as a screen shot in the next blog” so I think I was onto something.


The Wily charts have a “right-click” to copy to the Windows clipboard.  They are nearly useless, as no legends come with them.  A full window snap is more helpful.


I threw out a link to a blog I wrote on “time wasting loops” ( /people/jim.spath/blog/2009/03/24/sleeping-on-the-job) since part of the class exercise was to  find pseudo-bad-code.


Having 2 monitors, to keep up with the webcast slides/chat/votes, and the remote desktop, and a local copy of the slides, plus whatever other tools for reinforcement are needed, is a definite win.  A single laptop sized monitor would be awful (my original idea was to take this course in a hotel.  Probably would have been a big pain).


Adobe Web tool


Good:  The instructor learned he could “merge” people whose web sessions and phone lines were separated for some reason, but dragging one onto the other.  Pretty slick.


Bad: I hit the “X” by mistake and lost my session.  No “are ya sure?” or anything.  Gone.  Fortunately the phone line stayed on, though I lost my chat history and a few minutes of face time.





Marilyn Pratt asked again about alternative teaching styles.  I’m not sure I’d have the motivation to do all this reading, and the exercises, without the formal lecture time being carved out.  It’s certainly challenging for the instructor to maintain our interest, and stay away from reading slides.  I like the mixture of slide views intermingled with live demos and other feedback or exercises.




3 blogs on network troubleshooting (4th is in draft):



Assigned Tags

      1 Comment
      You must be Logged on to comment or reply to a post.
      Author's profile photo Somnath Manna
      Somnath Manna
      The instructor was starting to hint "I'd like to see this as a screen shot in the next blog" so I think I was onto something.

      Now you got the instructor hooked on SCN? 😀
      Always interesting to read your blogs even though technically they are often Latin and Hebrew to me.