Configuring and troubleshooting SPNego — Part 3

OK, now that we have installed SPNego with the Wizard (Configuring and troubleshooting SPNego — Part 1  (Configuring and troubleshooting SPNego — Part 1)) and did some basic checks on the client (https://weblogs.sdn.sap.com/cs/junior/view/wlg/8243  (https://weblogs.sdn.sap.com/cs/junior/view/wlg/8243)) it is time to take a closer look at the J2EE Engine. With the default log-configuration-settings only error messages are written to the log. So the first thing to get a more detailed look of what is going on is to increase the log level of the components involved with SPNego.

Some time ago you had to select each log location individually in the Visual Administrator, set the log level to “All”, reproduce the error, search the logs and set the levels back. Now we have two tools which allow us to debug all this much easier: the diagtool (https://service.sap.com/sap/support/notes/957666  (https://service.sap.com/sap/support/notes/957666)) and the web diagtool (https://service.sap.com/sap/support/notes/1045019  (https://service.sap.com/sap/support/notes/1045019)). Whereas the first tool has to be run and executed on the server, the Web diagtool offers the possibility to run and evaluate the traces from any client. Just deploy the ear files for that.

At first I will take a quick look (installation and start) at the diagtool, then continue with the web diagtool. In the end you get the results with both tools: detailed traces from which you should be able to see what is causing SPNego to fail.

Diagtool

The diagtool can be downloaded from here (

Note 957666 – Diagtool for Troubleshooting Security Configuration

). Just copy the file to your server and start it with 

go.bat confspnego.conf c:usrsap<sid>jc00j2eeconfigtool

Now answer the questions (ldap server, hostname, a windows username and password, Kerberos username) until you get to the part where you are asked to reproduce the problem.

Do it (access the portal from your client) and click OK. Now a ZIP file is created which you can take a look at (we will deal with that later on).

Web Diagtool 

The Web diagtool (Note 1045019 – Web diagtool for collecting traces) has the advantage, that you do not have to work on the server all the time. Just deploy the file web_diagtool.ear once (using SDM) and run the tool via http://server/diagtool  (http://server/diagtool).

Log Analysis 

Let’s take a look at the log files (I will concentrate on the log files created by the web diagtool, but all this information is also available in the diagtool). At first — like always — we will take a look at the scenario when everything is working.

Start the WebDiagtool and select security & all and click on  Go.

Click Add all.

Click Start.

Now try to log on to the server

and click on Stop and an overview page is displayed.

Click on Show All Traces (of course you can download a ZIP file version which you could sent to SAP support).

A page is displayed with general information about the system. You can also take a look at the login modules if you extend the login.modules (if you do that, you should see your login modules which can also be seen in the Visual Admin). Also all ume.properties can be displayed (but we will skip that for now).

From now on all the logs recorded are displayed. It can be a little confusing, but just scroll down until you see something spnego related (warning and error messages will be displayed in a different color and are easy to find).

Below you can see one example screenshot (keep in mind: my login was successful, but I still can see Warnings — so this is nothing to worry about).

The first important step for us is the step where the J2EE Engine is going through the SPNego template and is looking up the service user we created in the ADS as the very first step in Part 1 of this guide.  (“Received no SAPLogon Ticket. Authentication stack:  (spnego)” & “Looking for credentials for …”)

Here you can see that the J2EE Engine first checks if a SAP Logon Ticket is received (this is done because we have the EvaluateTicketLoginModule as the first module in our logon stack). This is not the case and the J2EE engine is now looking up the user defined in the krb5.conf file:

j2ee-j2e-vmw2053@DEV16.DEV-WDF.SAP.CORP

. Everything was fine (actually in this example the credentials were already cached, but you could see the acquisition in the Wireshark trace of the last blog) and the Credentials for realm DEV16.DEV-WDF.SAP.CORP were successfully acquired.

Now the J2EE Engine can continue with the next login module.

At first you will see an Access Denied error message. This is quite alright. As we could see in the HTTP traces of Part 2 the browser tries to access the J2EE engine not knowing that it has to authenticate itself. So no ticket is sent to the J2EE Engine right away and the SPNego login module cannot succeed. Only after this first call (and the received 401 — access denied) is the client requesting a Kerberos ticket for this server from the Ticket Granting Server. So the next access to the J2EE engine (just scroll down in the logs) looks much better:

This time we still do not have a SAP Logon Ticket, but a token (“YIIE…” — and we know from Part 2 that this is a Kerberos token) is sent along. The J2EE Engine can interpret this token and extract the servicePrincipalName (now knowing that this ticket is indeed for this J2EE Engine).

Also — just scroll down a little — the user credentials

YANGE1@DEV16.DEV-WDF.SAP.CORP

  are found in the ticket.

With this information the UME can try to lookup this user. Since we used prefixbased authentication the username is split in two parts (kpnPrefix,kpnSuffix) = (YANGE1,DEV16.DEV-WDF.SAP.CORP) and the search for this user is unique: Unique search result USER.CORP_LDAP.yange1

That’s it. Now the J2EE Engine can create a SAPLogonTicket and the next time we log on the EvaluateLoginModule will succeed right away!

Now to some examples when something is going wrong.

NTLM token received 

This is quite easy to detect. You can see that a NTLM token and not a Kerberos token is sent. Please make sure that all the browser settings are correct and that you access the J2EE Engine with an URL that is also registered as a Service Principal Name (e.g. if you have any DNS aliases — or even have added the servername to your local host file). For details take a look at the previous blog.

The error can also be “Error reading ASN.1 datastructure: java.io.EOFException”. That’s basically the same, just check the browser settings and the SPN.

Windows integrated authentication is not enabled

If you see only one call to the J2EE Engine (the access denied we talked about before), please check the settings for the Internet Explorer again. Make sure that Windows integrated authentication is enabled.

Clock skew too great (37)

When you take a look at the logs everything seems to be fine. But then — after the GSS Context created — you see an error:

GSSException: Failure unspecified at GSS-API level (Mechanism level: Clock skew too great (37)).

This means, that the time difference between the Client and the Server is to great (there is a default time difference for Kerberos which is usually about 5 minutes). Please check the time of both client and server. You can also try to issue a

net time /set /domain<br />

on the client (and on the server). It will syncronize the time from the client with the one on the domain.

Integrity check on decrypted field failed (31)

If you receive this error in the log please make sure that the password of the service user has not changed. If you are unsure, delete the folder kerberos (see Part 2) and set the username / password again. Make also sure that you selected Use DES encryption types for this account and run the Wizard all over. Before trying again clean your ticket cache (e.g. by logging of and on again). It could be that the ticket cached is wrong and you have to request a new one from the TGS (acutally this took me quite some time to figure out when I was writing this blog :-).

Acquiring credentials for realm failed<br />

Take a close look at the realm name. Is it really spelled correctly? What about uppercase / lowercase. This can also be a problem. In the logs you can see, that the realm the J2EE engine is looking for is in lowercase. But a closer look at the attributes of this user (remember: ldifde…) shows that it is in uppercase. Follow Note 1130190 – SPNego fails with “Failed to find any Kerberos Key” to correct this.

In gerneral: make sure that the data you enter is case sensitive. Take a look at the attributes of the users created and enter the data accordingly. A similar error can also appear if the Service Principal Name you entered (remember: setspn -A …) is missing or wrong.

Wrong JDK

Unfortunately I do not have a screenshot for this just yet, but if you encounter an error like:

Error in some of the login modules.

com.sap.engine.services.security.exceptions.BaseLoginException: Error in some of the login modules.

…Caused by: java.lang.NullPointerException

please make sure that you are not using Java 1.4.2_14, 1.4.2_15 or 1.4.2_16. All these versions contain a bug (Note 1057474 – NullPointerException in KRB5LoginMoule)

Thanks to the comments from Marc Lehmann and Hermann Hans to my first blog it looks like we have a workaround (at least for _15 and _16). Please try to add isInitiator = false to the com.sun.security.auth.module.Krb5LoginModule in the  com.sun.security.jgss.accept component via the Visual Administrator. I haven’t tried this out yet, but it seems to work!

!https://weblogs.sdn.sap.com/weblogs/images/37611/isInitiator.jpg|height=287|alt=image|width=600|src=https://weblogs.sdn.sap.com/weblogs/images/37611/isInitiator.jpg|border=0!</body>