Enterprise Portal time zone issue unfolded #sapadmin

[TomCenens]

Getting a proper understanding of the issue

An incident was raised by business that certain applications were randomly unavailable. Tickets were streaming in from different geographical locations. I work for the SAP expertise team which means I'm last in line do deliver support in case an incident takes place.

Whenever an incident reaches me it means I will have to get my spy kit and start investigating the incident thoroughly. Information is key and you have to have a proper understanding of the issue before you can tackle it.

With a bunch of incidents tickets I got a good idea which applications were unavailable and where the applications were residing. The technical infrastructure was pretty complex in this case spanning over multiple SAP and non SAP systems and a mixture of Windows and Unix servers.

In case you don't have enough information you should either seek out the information using the information you already have or you simply contact one or more end-users to get a better view on the issue. Once you have a clear view you move on to the next step.

From the point where you have a proper understanding what seems to be issue there are three important pointers to get to a solution:

• detect the actual root cause
• fix the root cause

Detect the component in which the issue resides

Detecting the component in which the issue resides can be done in several ways depending on which tools and applications you have available. E2E Root Cause Analysis in Solution Manager is designed to help you detect the component in which the issue resides.

In this case the component in which the root cause was situated was a SAP Netweaver Enterprise Portal 7.0 EHP1 which connects to the backend ERP SAP System. In the logging it was visible that there were authorization issues.

Detect the actual root cause

Because there were no clear error messages that provided me with sufficient detail I bumped up the log level to ALL in the log configurator for security related locations.

After doing so one of the log lines showed the following:

Logon Ticket has expired or is not valid yet. The ticket is valid from until and now it is . Authentication stack: tom.cenens3/blog/2011/07/19/enterprise-portal-time-zone-issue-unfolded-sapadmin/{3}.#4#Wed Jun 22 09:22:00 *EDT *2011#Thu Jun 23 01:22:00 *EDT *2011#Wed Jun 22 09:18:14 EDT 2011#ticket#

Now that might look fine if the server, the end-users and me where in the EDT time zone but it's not the case. I live in Belgium which is time zone CET, the end-users are in Belgium and the server is running in Belgium.

This explained how it was possible that the application was randomly unavailable. The EDT time zone is ahead of the CET time zone. This means that when the Single Sign On ticket was created in the morning it would only become valid in the late afternoon / evening.

Enterprise Portal time zone configuration

To find the actual root cause you first have to understand how the Enterprise Portal time zone configuration works. The time zone configuration is normally inherited from the Operating System for a AS Java based SAP system. In this case the system resided on Unix and the environment variable TZ is assigned to the <sid>adm Unix user which is the administrative Unix user which belongs to the SAP system with unique identifier <SID> (for example AA1).

You can check the time zone setting value through the system information page of your Enterprise Portal under Java parameter -Duser.timezone. In this case it was set to *America/New_York *which represents the EDT time zone.

The environment variable TZ was not active for user <sid>adm for some reason. The value is assigned to the variable when the users logs on in Unix. When there is no specific value assigned the first available time zone is applicable. So the Unix server was running in the correct time zone. A date command would give back the proper time zone and date / time. adm *user it had no time zone environment variable assigned at all which then means the value from a time zone configuration file is taken. The actual root cause appeared to be a file permission error on the file /etc/csh.login. This caused the Unix user <sid>adm not to receive the correct environment variable assignment.

Fix the root cause

To fix the root cause the file permission settings of the file */etc/csh.login were changed. To test I relogged the Unix user (using su - adm as root) and checked his environment variables (env*). The *TZ *environment variable now had a proper value assigned.

After this the SAP Enterprise Portal was stopped and started and the time zone configuration was corrected and with it the issues were also solved.

The parameter -Duser.timezone now showed *Europe/Paris *instead of *America/New_York *and the users were able to access the application again properly using Single Sign On.

An alternative that exists is to force the time zone configuration setting by configuring the parameter -Duser.timezone with the correct time zone in your Java virtual machine parameter settings. I don't recommend doing this compared to using the standard inheritance from the Operating System but it's nice to know it is also possible.