Configuring and troubleshooting SPNego — Part 3
OK, now that we have installed SPNego with the Wizard (Configuring and troubleshooting SPNego — Part 1 (Configuring and troubleshooting SPNego — Part 1)) and did some basic checks on the client (https://weblogs.sdn.sap.com/cs/junior/view/wlg/8243 (https://weblogs.sdn.sap.com/cs/junior/view/wlg/8243)) it is time to take a closer look at the J2EE Engine. With the default log-configuration-settings only error messages are written to the log. So the first thing to get a more detailed look of what is going on is to increase the log level of the components involved with SPNego.
Some time ago you had to select each log location individually in the Visual Administrator, set the log level to “All”, reproduce the error, search the logs and set the levels back. Now we have two tools which allow us to debug all this much easier: the diagtool (https://service.sap.com/sap/support/notes/957666 (https://service.sap.com/sap/support/notes/957666)) and the web diagtool (https://service.sap.com/sap/support/notes/1045019 (https://service.sap.com/sap/support/notes/1045019)). Whereas the first tool has to be run and executed on the server, the Web diagtool offers the possibility to run and evaluate the traces from any client. Just deploy the ear files for that.
At first I will take a quick look (installation and start) at the diagtool, then continue with the web diagtool. In the end you get the results with both tools: detailed traces from which you should be able to see what is causing SPNego to fail.
The diagtool can be downloaded from here (
Note 957666 – Diagtool for Troubleshooting Security Configuration
). Just copy the file to your server and start it with
go.bat confspnego.conf c:usrsap<sid>jc00j2eeconfigtool
Now answer the questions (ldap server, hostname, a windows username and password, Kerberos username) until you get to the part where you are asked to reproduce the problem.
Do it (access the portal from your client) and click OK. Now a ZIP file is created which you can take a look at (we will deal with that later on).
The Web diagtool (Note 1045019 – Web diagtool for collecting traces) has the advantage, that you do not have to work on the server all the time. Just deploy the file web_diagtool.ear once (using SDM) and run the tool via http://server/diagtool (http://server/diagtool).
Let’s take a look at the log files (I will concentrate on the log files created by the web diagtool, but all this information is also available in the diagtool). At first — like always — we will take a look at the scenario when everything is working.
Start the WebDiagtool and select security & all and click on Go.
Click Add all.
Now try to log on to the server
and click on Stop and an overview page is displayed.
Click on Show All Traces (of course you can download a ZIP file version which you could sent to SAP support).
A page is displayed with general information about the system. You can also take a look at the login modules if you extend the login.modules (if you do that, you should see your login modules which can also be seen in the Visual Admin). Also all ume.properties can be displayed (but we will skip that for now).
From now on all the logs recorded are displayed. It can be a little confusing, but just scroll down until you see something spnego related (warning and error messages will be displayed in a different color and are easy to find).
Below you can see one example screenshot (keep in mind: my login was successful, but I still can see Warnings — so this is nothing to worry about).
The first important step for us is the step where the J2EE Engine is going through the SPNego template and is looking up the service user we created in the ADS as the very first step in Part 1 of this guide. (“Received no SAPLogon Ticket. Authentication stack: (spnego)” & “Looking for credentials for …”)
Here you can see that the J2EE Engine first checks if a SAP Logon Ticket is received (this is done because we have the EvaluateTicketLoginModule as the first module in our logon stack). This is not the case and the J2EE engine is now looking up the user defined in the krb5.conf file:
. Everything was fine (actually in this example the credentials were already cached, but you could see the acquisition in the Wireshark trace of the last blog) and the Credentials for realm DEV16.DEV-WDF.SAP.CORP were successfully acquired.
Now the J2EE Engine can continue with the next login module.
At first you will see an Access Denied error message. This is quite alright. As we could see in the HTTP traces of Part 2 the browser tries to access the J2EE engine not knowing that it has to authenticate itself. So no ticket is sent to the J2EE Engine right away and the SPNego login module cannot succeed. Only after this first call (and the received 401 — access denied) is the client requesting a Kerberos ticket for this server from the Ticket Granting Server. So the next access to the J2EE engine (just scroll down in the logs) looks much better:
This time we still do not have a SAP Logon Ticket, but a token (“YIIE…” — and we know from Part 2 that this is a Kerberos token) is sent along. The J2EE Engine can interpret this token and extract the servicePrincipalName (now knowing that this ticket is indeed for this J2EE Engine).
Also — just scroll down a little — the user credentials
are found in the ticket.
With this information the UME can try to lookup this user. Since we used prefixbased authentication the username is split in two parts (kpnPrefix,kpnSuffix) = (YANGE1,DEV16.DEV-WDF.SAP.CORP) and the search for this user is unique: Unique search result USER.CORP_LDAP.yange1
That’s it. Now the J2EE Engine can create a SAPLogonTicket and the next time we log on the EvaluateLoginModule will succeed right away!
Now to some examples when something is going wrong.
NTLM token received
This is quite easy to detect. You can see that a NTLM token and not a Kerberos token is sent. Please make sure that all the browser settings are correct and that you access the J2EE Engine with an URL that is also registered as a Service Principal Name (e.g. if you have any DNS aliases — or even have added the servername to your local host file). For details take a look at the previous blog.
The error can also be “Error reading ASN.1 datastructure: java.io.EOFException”. That’s basically the same, just check the browser settings and the SPN.
Windows integrated authentication is not enabled
If you see only one call to the J2EE Engine (the access denied we talked about before), please check the settings for the Internet Explorer again. Make sure that Windows integrated authentication is enabled.
Clock skew too great (37)
When you take a look at the logs everything seems to be fine. But then — after the GSS Context created — you see an error:
GSSException: Failure unspecified at GSS-API level (Mechanism level: Clock skew too great (37)).
This means, that the time difference between the Client and the Server is to great (there is a default time difference for Kerberos which is usually about 5 minutes). Please check the time of both client and server. You can also try to issue a
net time /set /domain<br />
on the client (and on the server). It will syncronize the time from the client with the one on the domain.
Integrity check on decrypted field failed (31)
If you receive this error in the log please make sure that the password of the service user has not changed. If you are unsure, delete the folder kerberos (see Part 2) and set the username / password again. Make also sure that you selected Use DES encryption types for this account and run the Wizard all over. Before trying again clean your ticket cache (e.g. by logging of and on again). It could be that the ticket cached is wrong and you have to request a new one from the TGS (acutally this took me quite some time to figure out when I was writing this blog :-).
Acquiring credentials for realm failed<br />
Take a close look at the realm name. Is it really spelled correctly? What about uppercase / lowercase. This can also be a problem. In the logs you can see, that the realm the J2EE engine is looking for is in lowercase. But a closer look at the attributes of this user (remember: ldifde…) shows that it is in uppercase. Follow Note 1130190 – SPNego fails with “Failed to find any Kerberos Key” to correct this.
In gerneral: make sure that the data you enter is case sensitive. Take a look at the attributes of the users created and enter the data accordingly. A similar error can also appear if the Service Principal Name you entered (remember: setspn -A …) is missing or wrong.
Unfortunately I do not have a screenshot for this just yet, but if you encounter an error like:
Error in some of the login modules.
com.sap.engine.services.security.exceptions.BaseLoginException: Error in some of the login modules.
…Caused by: java.lang.NullPointerException
please make sure that you are not using Java 1.4.2_14, 1.4.2_15 or 1.4.2_16. All these versions contain a bug (Note 1057474 – NullPointerException in KRB5LoginMoule)
Thanks to the comments from Marc Lehmann and Hermann Hans to my first blog it looks like we have a workaround (at least for _15 and _16). Please try to add isInitiator = false to the com.sun.security.auth.module.Krb5LoginModule in the com.sun.security.jgss.accept component via the Visual Administrator. I haven’t tried this out yet, but it seems to work!
Thanks for this informative series of blogs on SPNego. If you set up a wiki, please let me know as I think that would be a great way of keeping up-to-date on that topic without having to search the entire SDN everytime 🙂
I have setup a Wiki Page in the Portal area. The "main" page deals with general information (https://wiki.sdn.sap.com/wiki/display/EP/Single+Sign+On+to+the+J2EE+Engine+from+Windows).
There is also a Troubleshooting section (https://wiki.sdn.sap.com/wiki/display/EP/Troubleshooting) where we could keep track of errors encountered during the installation.
Sincere appreciation for your blogs.
We have ABAP UME.
The wizard execution went well.
1.After the config was completed, We tried the kerbores authentication, it didnt work.
Not sure if the Kerberos ticket is even created.
Nothing comes up in Yatt and Wireshark.
The Yatt shows - message error 200.
Then I went through troubleshooting blogs 2 & 3.
a. I have a bit of confusion to understand...
According to Wizard for ABAP UME...
We will have for SPNEGO - Template (Policy Configuration)
EvaluateTicketloginModule : sufficient
SPNEGOLoginModule : Optional.
Hence in The Visual admin - It only shows
And it doesnt have entry for
SPNegoMappingLoginModule at all.
Then as per the Blog2
we changed the entries i.e
What should the entries be ??
Still no kerberos ticket,nothing comes up in Yatt.
did you check Configuring SPNego with ABAP datasource to configure ABAP UME with SPNego. Maybe this can help. If not come back and we will try to figure it out.
I will try the workaround which Marc Lehmann and Hermann Hans suggested and let you know...But it seems different people already have tested this workaround, and it seems to be working...Is there any reason why SAP officially does not accept this workaround as solution for the OSS Note 1057474
it is working (at least I have succesfully tested it on Windows with _16).
I am not sure why we do not mention it in OSS Note 1057474, but it could be that this is also a workaround by Sun-standards (if you take a look at http://java.sun.com/j2se/1.4.2/docs/guide/security/jaas/spec/com/sun/security/auth/module/Krb5LoginModule.html you will not find this parameter mentioned there).
To my understanding this is still a bug and has to be fixed.
I have also tested it with _17 in my Lab environment and it works fine.
As Marc Lehman have written Sun tries to make JDK 1.4.2_14+ more RFC 4120 compliant. In
Kerberos RFC 4120 I didn't see any
mentioning of the server
communicating/authenticating with KDC.
So it means that the server should not
login/authenticate against KDC. This can
be achieved by setting the parameter:
isInitiator = false
Which means that J2EE server will only
accepts the kerberos ticket, but not
Regarding the link you send...i think they did not updated the javadoc for 1.4.2 for a long time. This parameter is seems to be introduced starting from 1.4.2_14, but javadoc was not updated. Morover the javadoc for JDK 1.5 was also not updated...only in the javadoc for JDK 6 you can find the mentioning about this parameter, here is the link http://java.sun.com/javase/6/docs/jre/api/security/jaas/spec/com/sun/security/auth/module/Krb5LoginModule.html
Also i don't think Sun considers this as a bug (otherwise they would fix it already in one year and three JDK realeases after 1.4.2_14), I think they simply changed the implementation of Kerberos classes to be more Kerberos RFC 4120 compliant, and J2EE servers which only accept the Kerberose tickets should set the parameter as "false".
Anyway i opened OSS Message to SAP about this workaround, because we have problems in DEV and TEST systems for SSO with SpNego after upgrading of HP JDK (as per SAP EW recommendation)...and i want to apply this workaround to these systems. I will let you know the SAP reply.
I have the problem "KDC has no support for encryption type" and you mentioned a SAP Note 942111 but does not exist. Have you any ideas of this?
Thanks in advance.
unfortunately Note 942111 is no longer available. Please make sure that you do have the proper ServicePrincipalNames (setspn -a ...) set for the user.
I also encountered the same problem.
Do you know the content of this note - maybe you could send it to me?
did you use the Wizard? Can you check the service Principal name? Is the servicePrincipalName correct (with uppercase / lowercase?)
If it is not helping, sent me the diagtrace logs.
many thnx for your professional blog..
i have used SUN JDK 1.4.2_15..as per your blog
(com.sun.security.auth.module.Krb5LoginModule in the com.sun.security.jgss.accept component via the Visual Administrator)
i couldnt find that login module (com.sun.security.auth.module.Krb5LoginModule) in com.sun.security.jgss.accept. hol;ger how can i retrive this module?
if possible could you email ur mail id to me(firstname.lastname@example.org) whre i need your assitance in analysing the extract of the diag tool out put..
I have used Spnego Wizard for configuration, but I am still getting the below error message.
CreateContext failed: GSSException: Failure unspecified at GSS-API level (Mechanism level: KDC has no support for encryption type (14))
GSSException: Failure unspecified at GSS-API level (Mechanism level: KDC has no support for encryption type (14))
at java.security.AccessController.doPrivileged(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.AccessController.doPrivileged(Native Method)
The JDK is 1.4.2_15 and I have set the isInitiator to false.
System Details: NW 7.0 (SP 15) (ABAP + JAVA)
this looks like a problem with the service user you created. Please double check if DES encryption is set and the user is still valid.
If this is the case, either recreated this user and/or run the SPNego Wizard one more time (entering again username and password for the service-user)
I have run the diagtool and its shows no error/warning in LDAP settings?
Will a service user without DES Encription be shown as a warning/error message in the diagtool trace?
As LDAP configurations are not in my hand, changing thoes frequently becomes a cumbersome process 🙁
can you sent me the complete diagtrace to my email (from the Business Card?)
The user is still valid and I have run the wizard few times but the same error crops up.
Your part 1,2 3 blogs are very helpful. Now, we have made our SSO working fine for Single Domain. But we have a domain forest (Microsoft). Based on note 994791, it mentioned that for Domain Forest:
"Configure UME to use multiple ADS data sources (for each domain in the forest)".
I am not clear what steps are involved to configure UME for each domain in forest. Have researched in many places, and haven't found a good document for that (the one like your blog). Any helps would be appeciated.
there are differnt options to setup the UME to connect to a domain forest. It depends on how your ADS configuration looks like.
Feel free to contact me and we can discuss.
Thanks for the helpful blog.
I'm trying to setup Kerberos authentication, but I get a "Failed to find any Kerberos key" exception.
I've seen the relevant SAP note, and I've checked that the GSS name is correct, including case of letters.
To make sure the configuration is correct, I even ran the Sun Java GSS sample test programs (http://java.sun.com/j2se/1.5.0/docs/guide/security/jgss/tutorials/ClientServer.html), pointing them to the spnego wizard-generated krb5.conf file, and the tests succeeded. However, it doesn't work on the portal.
Do you have any idea what can cause the problem?
Server version: 7.0 SP9, JDK 1.4.2_12
without having the logs this is quite hard. Can you sent the logs to me?
Thanks for help
I do not fully understand: was it working with HTTP, but is no longer with HTTPS? - you should not have to change anything SPNego related for this. Just configure the J2EE Engine to use HTTPS via the SSL Provider Service.
If it is not working at all: are there any errors in the web diagtrace?
I try to be more clear:
1) If I connect to Http://myhostname.vvv.xxx:51000/irj no problem; i get into the portal without problems
2) If I connect to Https://myhostname.vvv.xxx:51001/irj i have to inser my user and password into the portal after pushing 'F5' to refresh the web browser.
In the log there is a 'authorization header' problem.
When i test in the config tool the ldap configuration, i have no problems. No problems also when i test the spnego wizard user resolution.
Do i have to make other setting when i use the https protocol to call the portal page initial page (after explicit log-on there is no problem).
this is quite strange. SPNego configuration is independent of whether you use SSL or not. So I guess this is somehow related to your SSL configuration.
Is it possible that you have a proxy that works differently when accessing this page via SSL? Do you have a any proxies in your landscape?
Is this happening for all users?
Could you run the WebDiagtool while connecting to the HTTPS portal? Do you see anything there in the logs?
I am getting the error in the title. As far as I am checking, I see nothing wrong in the browser settings. I have tried it both in IE8 and FF. The same is working for Win XP but not for Win7. We are not using any DNS aliases either.
I am configuring with the Spnego add-on wizard.
Any insight into this problem?
Thanks and regards,
I guess this is a known issue with Windows 7. Can you take a look at the following blogs:
SSO with SPNego not working on Windows 7 / Windows 2008 R2 (SSO with SPNego not working on Windows 7 / Windows 2008 R2)
New SPNego login module - just around the corner (New SPNego login module - just around the corner)
Thanks for the prompt reply.
I am aware of the blogs. In fact I have gone through them several times. They speak about enabling DES at the domain controller and rolling it out to the clients. This is not the ultimate solution as is pointed out by many. And hence is the new spnego add-on from SAP.
ADS for me is Windows 2008 server R2 and I am using the new spnego add-on.
I understand that this is a known issue- solutions are provided. Right now none of them are working out for me.
sorry, I thought you were still using the old login modul. The problem that you describe looks like still DES is used by the login module.
Can you then make sure that the service user that you are currently using does not have the flag "Use DES Encryption" enabled? Then create the Keytab file manually. You can also use Klist to verify that the created keytab file does support RC4-HMAC.
I would recommend that you also read through the comments in blog New SPNego login module - just around the corner
They are extremly helpful and provide lots of useful information not explicitly mentioned in the blog.
Service user created does not have the flag "Use DES Encryption". Keytab file was generated using KTPASS. The keys that I imported as part of the (new spnego Add-on)wizard based configuration were RC4-HMAC(code-23)and DES-CBC-MD5(code-3) respectively.
Your blogs are my only precise point of reference due to the contents and the comments. I am definitely following the comments!
Feel free to send me an email and we can discuss offline.
I keep on getting the below error, no matter what I do. The same set up worked fine in our dev portal, but QA portal is giving us this error even after so many trials. Please let me know if there is anything that I am missing.
Failure unspecified at GSS-API level (Mechanism level: Integrity check on decrypted field failed (31))
Portal: 7.01 SP7
after a migration of a portal system (source running on AIX/Oracle, target on SLES10/DB2) I'm facing the problem, that only a NTLM Token is send to the portal.
We set up a new technical user for the migrated portal and we also set the SPN of this user to the new hostname/dns-alias.
The customer is able to access the "old" portal with sso but not the new one. from the same client, so the client/browser settings should be ok. Do you have any idea why no kerberos ticket is sent to the new portal?
Many thanks in advance!
In our Test portal,SP Nego configuration is not working although its working fine for Dev Portal when tested from the same client workstation.
After running Web Dialog tool and Yatt tool ,it was found out that NTLM token is found in authorization header instead of SP Nego which is why its not working.
Also checked the Service User(j2ee-TJ1) in ADS ,its a unique user in
ADS and also the SPN for Test Enterprise pOrtal is tagged to only one Service user(j2ee-TJ1).
had refered both the SAP notes - 934138 and
1313880 and it seems okay ,also we do not have any DNS alias for this Portal.
I have had run the Kerbtray tool but its not much of useful and I am not getting the entry HTTP/yourServername when following the steps mentioned by you in your Troubleshooting Blog-2.
I found an error while running Wireshark - its showing Kerboros error -
I would appreciate any help in this
Its resolved ,there was problem with the service user which we were not able to detect even with ldifde commands,created a new user and it worked
SPNEGO configuration was already working but now all of sudden we started receiving errors in web diagtool:
Could not validate SPNEGO token. Reason: Next element is not type:32
and authentication does not succeed anymore. Did anybody witness similar error before?
Hi urosK, <br/><br/>I have seen a similar error some time back and it turned out that a user with the same service principal name was created in the AD. Can you run the ldifde command ldifde -r (serviceprincipalname=HTTP/<server.domain>) -f spn_output.txt<br/> (like outlined in Configuring and troubleshooting SPNego -- Part 2) and check if that is the case. <br/>If not, then I would recommend to create a CSN / OSS message and attache the output of a WebDiagtrace. <br/><br/>Regards, <br/><br/> Holger.<br/>
What a fantastic resource - I truly couldn't have got our SSO working without it!!
Just one ommission that might be worth mentioning in there somewhere. The last blocker we had to getting SSO working reliably across our domain was that fact that 2 of our six domain controllers were running windows server 2008R2.. the others were all 2003. DES encryption is by default disabled on 2008 & r2 as it isnt deemed that secure anymore. it seems that kerberos needed DES before tickets would be issued. Anyway, enabling that sorted us out.
Many thanks again for this great resource.
For all those thinking: Does the webdiagtool still work in 7.3 or 7.4 ?
Yes, it does. 🙂
I just deployed the 7.1 version via telnet and deploy command.