This blog is part of the #SAPADMIN launch, an idea that was launched to have a space for system administrators on SCN. You can read the leading blog here. Martin English made a dummy concept page which you can check out. Also check today’s blogs for hash tag #SAPADMIN and check twitter for tweets that are tagged with #SAPADMIN. You can find the idea on idea place here.
I have been working on the root cause analysis some time now and while I love the product I still see a lot of room for improvements. I thought it would be nice to share some tips, tricks and write about the possible improvements.
The tools surely has its benefits, tracing HTTP content (Portal, Web Dynpro) to identify the component where the bottleneck lies gives can give nice results like picture 1.1 where there was a network issue at the time I performed the trace. If the tools would be not be useful I wouldn’t frequently blog about it.
The Diagnostics Agent is used in the root cause analysis scenario (also called diagnostics). In short, the agent fetches data and sends it to the SAP Solution Manager where it is stored in BI Infocubes.
The installation of diagnostics agent 7.20 is performed using a SAR file package which features a SAPinst once unpacked that you can use to install your diagnostics agent. You also need the JCE Policy 6 zip file as Diagnostics Agent 7.20 is using SAPJVM 6.1.
Host Agent included
The Host Agent is used to provide services that can be accessed by the SAP systems residing on the same host. For diagnostics the main service provided is SAPOscol (SAP OS Collector) which collects OS statistics (CPU, Memory, Disk etc). The Host Agent is now automatically included in the SAPinst provided to install Diagnostics Agent 7.20.
Mini SAP system
The Diagnostics Agent consists out of a mix of different processes that existed already. You can see it as a mini SAP system. Where in the past you had so use scripts to stop/start the agent you can simply use the standard commands startsap and stopsap now. The build of the agent also looks very similar to a regular SAP system with profiles, kernel and so on.
The agent has a toned down SAP kernel present which consists out of a number of processes that are part of the package SAPEXE. Besides that you also have a SAPJVM 6.1 package as the Diagnostics Agent is running on Java technology.
Update update update
Some time ago a blog was posted by Karol Frankiewicz with information how to patch the SAPEXE to avoid errors upon startup. What happened is that at startup the profiles are checked and this would throw messages that there are missing parameters as the startup process thinks that the Diagnostics Agent is a SAP system which has an ICM on board. To get rid of these errors you can update the kernel files that are used by the agent.
I mailed Karol and told him the instructions are not all that clear and I was happy to see he responded very quickly and stated he would change the instructions based on our conversations through mailing. So you can expect to see more clear instructions on the kernel updating of the agent in SAP Note 1466109 – Support of system/type=SMDA in the SAP profiles.
The kernel isn’t the only package you should update, the SAPJVM 6.1 and the Host Agent should also be updated but more about that later in this blog.
Since I do have other points which I wanted to address I decided to write this blog and give feedback to the SCN community members to improve the installation process in order to avoid as much issues as possible afterwards.
There are definitely some pain points performing the installation using the regular SAR file provided. During the installation you can specify a kernel package and a SAPJVM6.1 package that you have downloaded. This means the installer will then use these files instead of the standard files that are delivered with the installation DVD.
There goes the free disk space
Don’t be surprised if you run out of disk space as the SAPEXE package that is delivered with the agent only holds around 78 files and the SAPEXE you download from marketplace (mentioned in the SAP Note to update the Diagnostics Agent kernel) holds at least 150 files.
Another problem is that the SAPJVM 6.1 package you downloaded will be installed along with the initial version. So you will have a SAPJVM 6.1.001 in your file system sitting and doing nothing and a SAPJVM 6.1.<version you downloaded) which is used. I really don’t want that SAPJVM 6.1.001 consuming disk space.
Shut it down
The Host Agent install is done automatically but it also means you run into problems if you install a second Diagnostics Agent 7.20 on the same host as your Host Agent process will be running and cannot be overwritten by the automatic installation so you will have to temporarily shut it down.
This also means that the Host Agent files get overwritten in the process so hopefully you didn’t update it manually yet because you will have to update it again in that case. Another possibility is of course that you edit the XML file and skip this particular step. You can read more on editing XML files for SAPinst in one of my previous blogs.
How can these issues be solved?
The proper solution is most likely to rewrite the SAPinst procedure for the Diagnostics Agent. There should be a check if the Host Agent is already installed and if yes give the option to overwrite or keep the current version. The SAPJVM 6.1 should not be safeguarded in my opinion as it’s an initial installation and the old version will only consume disk space for no good reason. The kernel files that are necessary should be copied out of the provided SAPEXE SAR and not all the kernel files.
The only good solution at this moment I see to avoid as much of these pain points as possible is to remodel the installation DVD with the latest available version of those components. In order to do that you need to find the files of course.
Remodeling the installation DVD
Where are the damn files at
It can be a challenge sometimes to find files on SAP Service Marketplace, I’m sure many can relate.
To be able to update you need to find the files first of course. It seems there were many questions on where to find the files because SAP Note 1442124 – How to download a SAP JVM patch from the SMP states where to find the SAPJVM files. Don’t forget the V=MAINT at the end of the URL as it doesn’t come along if you copy paste the URL from the SAP note.
The instructions to manually update SAPJVM can be found in SAP Note 1025085 – How to manually patch the SAPJVM.
SAP Host Agent
You can find the SAP Host Agent by following the path on https://service.sap.com/swdc -> Support Packages and Patches -> Browse Our Download Catalog -> SAP Technology Components -> SAP Host Agent.
Updating the Host Agent is easy and goes quickly. You can find instructions in one of my previous blogs. The instructions to update the Host Agent are still the same just make sure you use the right file version of course. If you bump into the issue that a file cannot be copied, copy it manually from your temporary directory into /usr/sap/hostctrl/exe and rerun the saphostexec –upgrade in your temporary directory to see it finish successfully.
You can find the path to the actual SAP Kernel by clicking on the correct version under the SP Patch Level of SAP Note 1466109 – Support of system/type=SMDA in the SAP profiles which was created to address the warning messages during the startup when using an older version of the SAP Kernel for the Diagnostics Agent.
When you unpack the SAR file delivered for the installation of the Diagnostics Agent 7.20 you will see two DVD’s, one which holds SAPinst and one which holds the packages of SAPJVM, the Host Agent and the SAP Kernel.
You can remodel the installation DVD by changing/replacing the SAR files contained in the Kernel installation DVD.
To change the Kernel files, unpack the SAPJ2EE.SAR to a temporary folder X, unpack the new SAPEXE kernel to a temporary folder Y. Create a list of files that are contained in folder X and launch a little script that loops through the list and only copies the files from folder Y to folder X that are contained in the list. That way you are sure you have all the necessary files and as much as possible is updated with the latest kernel version.
An alternative described in the SAP note (unpacking into global, syncing global to local by starting the agent, stopping the agent, cleanup global exe and copy local exe to global exe) but this requires you to have enough space available to perform these operations. Global exe dir is /usr/sap/<SID>/SYS/exe/uc/<platform>/ and local exe dir /usr/sap/<SID>/SMDA<instance no>/exe.
The SAPJVM6.SAR files is easier, download the latest SAPJVM 6.1 version and rename the SAR package to SAPJVM6.SAR and replace the original SAPJVM6.SAR of the kernel installation DVD.
The same goes for the SAPHOSTAGENT.SAR file, download the latest available version, rename and replace the original SAPHOSTAGENT.SAR of the kernel installation DVD.
SAPOscol errors in the Agent log files
The Diagnostics Agent 7.20 looks for the process saposcol in /usr/sap/<SID>/SMDA<instance number>/exe and if it’s not found throws an error in the diagnostics agent log file. While it would not be a problem as the Host Agent is running SAPOscol as a service and it should be accessible and you should also see data appearing in Wily Introscope, you can get rid of the errors by copying the saposcol that is located in /usr/sap/hostctrl/exe to the path /usr/sap/<SID>/SMDA<instance number>/exe/.
If tenured space usage is not showing create a customer message
The troubleshooting guide which is available on SAP Service Marketplace has some interesting content but in my opinion more should be added to the guide. SAP support had many messages according to what I have seen and they should already have an extensive knowledge base on possible problems. Surely they have content on what to check if the tenured space usage isn’t showing in Wily Introscope. It would be a good idea to update the troubleshooting guide with as much information as possible to enable to customer to do more initial searching and possibly solving the issue without the need to open a customer message.
In my opinion customers implement the scenario and often don’t really know or realize something is missing since there is no alerting that says “hey you are missing something” they assume everything works as it should be but in fact often there are metrics missing.
Pain points for the implementation of the root cause scenario
I have been working with root cause analysis some time now, in the beginning there was a lot of frustration, today there still is some frustration but slowly it’s getting better, bit by bit. There are still some major paint points when you have to implement this at a customer site. If the prerequisites are not fully met expect to have a 90% working scenario or even less.
While you could say of course you have to meet the prerequisites it’s not always feasible in a short amount of time. The problem starts with the correctness of the system landscape in transaction SMSY in SAP Solution manager.
In general the transaction is poorly maintained by most of the customers, do I blame the customers then? Not at all, the way the transaction is build and the way the data is updated is not good enough.
The recommended way is to have the managed SAP system push data to the SLD of the Solution Manager. Directly or through an SLD bridge. That data is then fetched in the Solution Manager from the SLD to fill transaction SMSY. However, if you have ever set the data fetching to use method RFC/TMS the data is not updated automatically when the fetch takes place (paint point right here). Clicking on a button to read the system number of the managed SAP system is enough to change the fetching method from SLD to RFC/TMS. Changing it back is not possible, you would have to delete the SAP system and recreate it through a SLD data fetch.
The job that fetched that data titled LANDSCAPE FETCH which you can find in transaction SM37 can have seriously long runtimes. Two thousand seconds is value I have seen frequently. You can expect to see a change in this mechanism in Solution Manager 7.1 though but I don’t know to what extent they will actually fix the other problems.
The only way to get the data corrected properly is to delete the managed SAP system in transaction SMSY along with the data that belongs to the SAP system (database, java instance). Again pain points here as the SAP system might be included in numerous SAP Solution Manager projects which are then again used by other scenarios such as CHARM (Change Request Management).
Fixing the content would then mean removing the SAP system out of every scenario that is in use. Are you serious? Yes. It doesn’t stop there because if the product assignment of the SAP system is incorrect you are soon looking at changing the product assignment of all the SAP systems contained in the logical component in order to switch the logical component to the correct product. Take into account the fact that you have to wait for two-thousand seconds when you perform the change.
How do you get rid of inactive old agents
You have two different locations were your old agents might reside. One is among the offline agents in the agent administration application (accessible through root cause analysis workcenter). You can see in picture 1.2 there 14 agent(s) offline. Some or all of them are obsolete, old and can be removed.
To remove the offline agents which are no longer running or required first click the show offline agents’ button to show in the list of agents. Next click on the Filter on button and type a filter to have one or more agents remaining in the list (the agents you wish to remove). Next hit the Remove offline agent entries (see picture 1.3).
Agent Candidate management
There is another location in the agent administration application were you could have obsolete entries. The Agent Candidates Management (see picture 1.4).
Click on the Agent Candidates Management link and next choose custom SLD Connection Parameters (see picture 1.5). Maintain the parameters listed below and click the Apply button. You will then see the agents that are connected to the SLD which were once inserted into the Agent Candidates. Newer agents register directly if you keep default settings during installation and they will not appear in the list.
Now there is no remove button here but you could have already noticed the data is being read from the SLD so you can remove the obsolete agents in the SLD.
In order to remove obsolete agents from the SLD logon to the SLD using URL http://<host>.<domain>:5<instance no>00/sld with a user that sufficient rights to administrate the SLD.
Next go to Administration (see picture 1.6).
Next click on Content Maintenance under header Content (see picture 1.7).
Now from the dropdown boxes choose All Classes for Subset and Diagnostics Agent for Class as shown in above screenshot (see picture 1.8).
This will provide you with a list of agents that exist in the SLD (see picture 1.9). Next to the above shown columns you have hostname reference and more details (since it is sensitive information I cannot show it). To remove an agent from the SLD, mark the checkbox in front of the agent and push the remove button which is available in the table toolbar.
Overall there are some improvements compared to the previous version in terms of installation and configuration, you can directly register your Diagnostics Agent which takes away the need to go through yet another registration process on SAP Solution Manager. Instead you can now directly start using the managed SAP system setup once your agent is installed. SAPinst also checks the users and passwords entered during the installation for correctness which is also useful as in old versions you only noticed once the installation was finished.
Concurrent mark sweep interpreter available
The interpreter which was missing for the Concurrent Mark Sweep garbage collection is also ready and my customer message was answered by development to let me know the following versions support the garbage collection output:
Support of CMS GC output is now included in SolMan 7.01 with
– LMSERVICE SP07 Patch 3
– LMSERVICE SP08 Patch 1
and SolMan 7.1
– LMSERVICE SP01 Patch 1
Support list is an extract from a SAP customer message
I have to state that I’m getting the necessary support from by SAP so far help solve issues and that feedback is well received and actually being handled. This shows in a number of notes, documentation and patches.
It’s important to give feedback to help improve products. I’m glad to see SAP is willing to listen and work together to further enhance the product. Of course they said themselves they want more open communication and a better connection to the customer so this fits right into their recently announced strategy.
You can expect to read more on this as improvements are made and pain points are being handled.