Configuring and troubleshooting SPNego — Part 2

In the Configuring and troubleshooting SPNego — Part 1 I explained how to configure SPNego with the help of the SPNego Wizard. Now, that everything is set up we are going to check the configuration. At first I will simply use some tools (see below to get a summary of tools used) to “verify” the configuration, e.g. I will use the tools but the results will always be OK. I think it is always good to have one working example and then see what is different / what it should look like if you acutally have a problem.
In this blog I will not use the diagtool or Web Diagtool. Instead I am going to concentrate on some basic things you can check on the client before using these tools. So let’s get startet…

For all the screenshots and description I have used the same setup as in Part 1: my user is yange1 and the server is installed on the server vmw2053.wdf.sap.corp.

The server side

First let me show you what the SPNego Wizard has done. Start the Visual Administrator and take a look at Server -> Services -> Security Provider. A new component com.sun.security.jgss.accept was created which contains two LoginModules: Krb5LoginModule and SPNegoMappingLoginModule. Both contain the options you chose when clicking through the wizard. Among other settings the Krb5LoginModule contains the properties of the Kerberos Principal user (service user) and the SPNegoMappingLoginModule the user resolution mode.

Then there is also the spnego template which contains the login modules required for a succesful login. The first entry is the EvaluateTicketLoginModule (com.sap.security.core.server.jass.EvaluateTicketLoginModule). The Login module checks whether you already have a valid SAPLogonTicket (in a federated portal scenario this ticket could also come from another portal and if you have chosen to trust this portal then the check would succeed and because of the Flag “Sufficient” you would simply skip the next modules). If the Evaluate did not work, then the next login module will be used: SPNegoLoginModule. This module does the actual SPNego/Kerberos check. The flag is Requisite so that if it succeeds it will continue with the next login module. When this login module was successful then the last login module CreateTicketLoginModule will be executed. This time a SAPLogonTicket will be created so the next time you query the portal the EvaluateTicketLoginModule will succeed right away. (For a more detailed list of what the different Flags mean, please check: http://help.sap.com/saphelp_nw70/helpdata/en/d0/ee244134a56532e10000000a1550b0/frameset.htm)

On the file system a new folder has been created under usrsapSIDSYSglobalkerberos. This folder contains the keytab file which allows the J2EE Engine to access the service user (you created in the very first step in the LDAP directory) and the krb5.conf file which contains all the information about the Key Distribution Center, the Realm and the encryption type.

Maybe you have started the SPNego Wizard a second time and were wondering why only four configuration steps (instead of five) were available. Just delete the folder kerberos and you can (almost) start from scratch.

If you start the config-tool you can see that the Wizard did also add some Java start-up parameters to the server node. Among them is the location of the krb5.conf file.

One last thing (that was not done by the Wizard, but by you) is the dataSourceConfiguration file. This file (if you are using the Microsoft ADS variant) is just an enhancement of the default dataSourceConfiguration_ads_readonly_db file that comes with the J2EE Engine. The new file includes new attributes which are used by SPNego (e.g. krb5principalname, kpnprefix, dn).

The Client side

First of all log on to the client and check if you have Kerberos tokens which can be sent to the J2EE engine. In order to do this, download the tool kerbtray from Microsoft. Kerbtray is part of the Windows Resource Kit and can be very helpful.
After running the tool you can see a small green icon in your task bar. Simply right-click on it and choose List Tickets. You will see your Client Principal in the first line. This name will be used for the lookup in the UME (in the screenshot it is YANG1@DEV16.DEV-WDF.SAP.CORP).

Before explaining what is done with this ticket we want to check that a Kerberos ticket and not a NTLM ticket is sent to the J2EE Engine. The easiest way to do this is to use a HTTP Tracer like HTTPWatch. Since HTTPWatch is not freeware you can also use other tools to collect HTTP traffic (if you are going to send log files to SAP Support you can still use this tool and save the full trace which can be evaluated by SAP). A very nice (freeware) tool is YaTT. This tool allows you to easily trace the traffic. Just download it from here and install it on your client. Now start the tool and enter the URL of your J2EE engine as a Name Filter (you can also specify the port).
The tool is collecting the TCP trafic to the J2EE Engine. So start a new browser and access the URL. The first request you will see is a 401: Access denied — which is good! The J2EE engine tells the browser that it requires some kind of authentication. Now the Client can check the Ticket Granting Server (TGS) if a ticket for this engine is available (we will come to this later). When this check was successful the client can sent a Kerberos ticket to the Engine. This ticket can be identified by the starting string YII… [NTLM starts with TIRM…].

Now that we have accessed the J2EE Engine check the Kerbtray tool again (in order to see any changes, you have to close and open the tool). You should see a new entry HTTP/yourServername. This means that your Client has requested a ticket for the J2EE Engine on this servername, which it then could provide for authentication.

This ticket contains your user name (as seen before YANG1@DEV16.DEV-WDF.SAP.CORP). Since I have chosen prefix based in my SPNego configuration the name will be split in KPN-Prefix=YANG1 and KPN-Suffix=DEV16.DEV-WDF.SAP.CORP and the J2EE Engine will try to search for YANG1 first and if this is not unique will use the suffix information to get a unique user.

That’s it. Now you should be logged in.

Provided that everything is going well, we have the above scenario. However, there are plenty of pitfalls that you might encounter. One problem I encountered pretty often is, that a NTLM ticket but not a Kerberos ticket is sent. This can have plenty of reasons and the first step would be to check help.sap.com
* Enable Windows Integrated Authentication in your Web browser.
* Enable automatic logon in Intranet zone.
* Add the AS Java’s DNS host name to the list of local intranet sites.

If you did all these checks and Kerberos is still not working try it from another client (just to be sure). Then check Note 934138 where some issues with (missing) Bug-Fixes from Microsoft are deal with (Hotfixes 885887 and 899587 from Microsoft).
If it is still not working, then you will have to do a more detailed traffic trace. I will cover this in my next blog. 

If no ticket is sent at all then you might have to check whether the client is able to find the J2EE Engine in the ADS. In order to setup SPNego we had to create a service user and set certain Service Principal Names to it. If there was a typo or if the name is not unique the lookup will fail. So use the next tool also available from Microsoft (it may be already installed): ldifde. ldifde allows you to query the ADS and search for any attributes.
At first I will try to find the user I created for SPNego. This can be done by issuing the command

ldifde -r (samaccountname=j2ee-j2e-vmw2053) -f sam_output.txt

-r means that ldifde is searching recursivly and -f specifies a file into which the results are written (see http://support.microsoft.com/kb/555636/en for further details).

If you take a look at the output file you will (hopefully) see entries like:

sAMAccountName: j2ee-j2e-vmw2053
userPrincipalName: j2ee-j2e-vmw2053@dev16.dev-wdf.sap.corp
servicePrincipalName: HTTP/vmw2053
servicePrincipalName: HTTP/vmw2053.wdf.sap.corp
 

Now that I know that my user was created (and I can see that the ServicePrincipalName (SPN) are set and correct) I want to make sure that the SPN is unique. So I am going to issue the command:

ldifde -r (serviceprincipalname=HTTP/vmw2053.wdf.sap.corp) -f spn_output.txt

It is important, that you will only get the result “1 entry exported“. If you get more than one entry, then open the file and see what other user is using this SPN.
(In order to get other information about a user in the ADS you can use a LDAP browser like Softerra. It is free and I find it very helpful to get a quick overview of your LDAP structure and the user configured.)

You might also have a problem, that you are accessing the portal not with the real physical hostname, but with a DNS alias. To check whether you have really added all relevant SPNs to the user do a nslookup on the servername. Make sure that all listed aliases are also added as a SPN.
nslookup vmw2053.wdf.sap.corp

If it is still not working …

Well, then we should take a look at the Kerberos token acquisition. For this we are going to use Wireshark. Just start Wireshark on the client and (in order to decrease the amount of data being displayed) set the filters: ip.addr == YourKDCName.
Again, I will show screenshots where the acquisition is working. In the final blog I will deal with several issues by looking not only at the Wireshark traces, but also the logs from the J2EE engine.

Start the trace (Capture -> Start) and access the J2EE via the browser.
The first thing you will see is a TGS-REQ, where the client connects to the ticket granting server to request a ticket for the J2EE engine. The J2EE Engine is known to the TGS only via the URL you are accessing the Engine: http://vmw2053.wdf.sap.corp => HTTP/vmw2053.wdf.sap.corp. This can be seen in the trace:

The next step is the response from the TGS TGS-REP: it contains the ticket which will be sent to the J2EE engine.

Here you can see the client name (this is the user that logged on to the client) and you see the ticket in the response. This is the ticket for our user and the server which we requested the ticket for.
This ticket will now be sent to the J2EE engine and from there the client name will be used and extracted (yange1@DEV16.DEV-WDF.SAP.CORP) for authentication.

You can also run the Wireshark on the server. The J2EE Engine has to acquire a ticket for the service user. That’s why a firewall cannot block the access to the KDC from the server. In the logs you can see the request for this user first:

Then the KDC sends the response and this ticket is cached by the engine:

OK, that’s it for the second part. In the next part I will take a look at the log files from the J2EE engine. By increasing the log level you will get valuable information about what errors are causing SPNego to fail. 

Tools used so far

• Kerbtray: http://www.microsoft.com/downloads/details.aspx?familyid=4E3A58BE-29F6-49F6-85BE-E866AF8E7A88&displaylang=en
• Yatt: http://www.pocketsoap.com/yatt/
• Softerra: http://www.ldapbrowser.com/
• Wireshark: http://www.wireshark.org/
• ldifde
• nslookup