Ok,  I haven't posted anything since July of 2008 but I think I have some reasonable excuses.  The first problem was a real lack of content to contribute.  My last 10 months has been a lot more about internal solution assessments (more to come on what those are later) and working on ramping up to handle support issues when they start to flow in.  Well that time has come,  I am now working multiple issues and I have been doing lots of troubleshooting over the past 2 months.  This is creating great blog material so if I stay disciplined I'll start writing up an entry every week or two.

Now onto the real meat of this post.  Let's talk about Kerberos,  you know the thing you always hear can be used to do end to end delegation of user credentials but when you try and set it up something always blows up and you revert back to NTLM and use the trusted sub system model.  At least that's what I've seen and done myself on a number of occassions.  Well,  that doesn't work real well when you get into a scenario where a user is integrating SQL Server Reporting Services reports into MOSS.  Add to that the desire to connect from both domain joned and non-domain joined machines and you get a whole bunch of variables that you have to mentally juggle as you are troubleshooting the dreaded 401 access denied HTTP error.

Let's start with an explanation of why Kerberos is really your only good option when setting up MOSS and SSRS integration.  If you haven't worked with this yet it really is a very slick way to provision your reports and even your report builder tools out to your users.  When you install SSRS (SP2) you will get the option to setup your SSRS enviornment in integrated or native mode.  The Integrated mode is where you can actually hook into MOSS.  You'll also need to install some SSRS bits on the MOSS servers as well and you'll need to activate the reporting services content types that will now be available in your document libraries.  Once you do all that (there's more detail to it but this is not a SSRS/MOSS integration deployment post) you can add a document library and then add new reports, report data sources, and report data models.  Now you have a central location for your users to create and review reports.

What does this look like from a security perspective? First, you have a user coming from a client machine and authenticating to a MOSS site.  Once they do that and click on the doc library where SSRS is called you have to make calls over to the SSRS server to get the data to display or options for the various controls on the content type forms.  So what options do you have for setting this up,  well you have Windows Authentication and Trusted Account as options in the SSRS integration settings.  What those essentially translate over to are impersonate = true or impersonate = false.  If you want to make your life easy here you can just setup trusted account and make sure the MOSS applicaiton pool account has access to all the databases that the reports need and you're good to go (SSRS is setup to impersonate by default). Wait a minute ... ewwwwwwww.... that sure sounds like a "super-account" to me.  You probably do *not* want to do that if you ever plan to have sensitive reports access sensitive data (which you inevitably will).

That brings us back to "windows authentication" which means we're flowing credentials from the users.  Unfortunately,  our good old stand by of just use NTLM challenge/response if Kerberos is too complicated to get setup is not going to work.  It's well known that NTLM has a by design limitation that doesn't allow it to go over 2 physical hops.  That means when a client has their credentials passed through to MOSS that then passes it along to SSRS that then passes it along to SQL Server or any other database that supports integrated auth you are dealing with three physical hops.  Of course you could use username/password authentication to avoid having to flow user credentials to the database but that is also not a great security practice.  Other options do exist within SSRS to run the report under different credentials and you could technically maintain multiple reports with different "run as" credentials and secure them in sharepoint with different doc libraries but what a maintenance nightmare that would be.  You can probably tell I went down just about every path I could think of here because of the frustration of getting Kerberos working end to end.

One last option that was explored that I want to mention is the ability to use different security zones in MOSS.  This is a very powerful feature that allows you to have different web sites share a content database in MOSS but use different authentication providers.  I did in fact explore a temporary work around that would use NTLM for intranet users and Basic Authentication for non domain joined users (with valid domain credentials).  This would require SQL Authentication as well to avoid a physical hop to the DB from the report but would allow the two different groups to access the reports differently.  In the end this was not ideal because the better practice is to ultimately remove sql authentication and use integrated authentication and flow the credentials.

So I was back to the drawing board and it was now time to really figure out how to make Kerberos work.  The first thing that you'll find difficult about dealing with Kerberos is understanding whether or not it is actually working.  The operating system is very good at failing over to NTLM if any thing is setup incorrectly with your Kerberos request.  The first thing you need to get your hands on is the Windows Server 2003 Resource Kit so you can get kerbtray and klist.  This will show you the tickets that are successfully provisioned via the Key Distribution Center (KDC) which is the manager of Kerberos ticket brokering from your domain controller.  You will also need to become good friends with the registry, event log, and lsass.log file.  There are a lot of trace switches available for Kerberos protocol traffic that can help you diagnose what is going on.  Here are a list of the four steps I followed to get more detail:

1) The first debugging step for something like this is to do a netmon capture (http://www.microsoft.com/downloads/details.aspx?FamilyID=f4db40af-1e08-4a21-a26b-ec2f4dc4190d&DisplayLang=en) and look at the packets going back and forth as a repro of the issue is performed.  We did this in my lab enviornment and  I could see plenty of KerberosV5 tickets with the SPN requests and whether or not they succeeded.  That didn't however prove to be successful when looking at captures that came in from the environment where the problem was occurring.  Here was my first lesson ... if you really want to know what is going on you *must* get a capture from both sides of the communication.  I receieved numerous captures from the first server the user was trying to connect to and never found any KerberosV5 packets.

2) Next I turned on event logging on my application servers that were creating the kerberos tickets.  The following link gives detailed information on how to do this: http://support.microsoft.com/kb/262177 .  The key thing you get here is detail when things fail over to use another protocol which just magically happens when you don't have something configured correctly. 

3) The next troubleshooting tool I found for this was actually a website that could be setup in IIS to check all Kerberos settings to point out any setup issues.  You can download that here http://www.iis.net/downloads/default.aspx?tabid=34&g=6&i=1434 .  Just set this up in a website using the same app pool identity (not the same MOSS site you were currently using a different site using the same app pool) and give it a try logging in with your user account.

4) The lowest level of tracing/logging I pulled together was the logs from the lsass.exe process.  There are actually a number of levels here that you can pull in to determine what types of messages are being passed back and forth between the client and the KDC.  To enable this follow these instructions (scroll down to "Kerberos"): http://blogs.technet.com/askds/archive/2008/04/02/directory-services-debug-logging-primer.aspx . This one is going to be tough to do if you don't have a virtual enviornment to reproduce the configuration.  In a typical enterprise there are typically no test/dev domains that can be tinkered around with so making registry edits on the production domain may be a total dead end for you.  In my case,  I was able to use a single machine with WIn Srvr 2008 and Hyper-V with 8 GB of RAM to simulate this entire 4 machine configuration.  I highly recommend this as a way to validate all of your findings and experiment with any changes before you start to look to make changes to the live domain.

Ok,  what did all of these logs tell me?  I considered bringing in each of the different sets of detail logs into this post and show you how I analyzed the results but that would probably turn this blog entry into an epic (if it isn't already ).  I will just say that it did point out numerous problems with my Service Principal Names (SPNs) which are a constant source of headache when setting Kerberos up end to end.  This is especially true in the case where you have different identities running the various application pools that are attempting to delegate user credentials.  I have to say that's why I think this is so complicated.  You're dealing with four totally different concepts here.  Authentication being the first one.  How are you telling everyone involved in the security session who you are.  Then you get to Auhtorization or who is granting you permission to do what you're asking?  Then you have to talk about Impersonation which is whether or not the process you're working with allows you the caller to ask for stuff outside of its boundaries.  Lastly,  there's delegation which goes hand in hand with impersonation because it's about authorizing the service asking for permission to allow you to ask for other stuff.

Now that I had isolated the issue to the SPNs I could go back to my domain controller and use the setspn tool (available in the OS and can be run from anywhere on the domain but requires domain admin privileges to make any changes). The setspn tool allows you to add SPNs for a service account or a machine that will identify that account or machine when it is attempting to perform kerberos security ticketing.  Now there are multiple things I did wrong here when I was first trying to set this up.  The first one I did was setup the same SPN for all my service accounts (MOSS and SSRS) but that doesn't work because you can not have duplicate SPNs setup on multiple accounts.  Unfortunately,  I was only differing my sites based on port number inside of MOSS and I needed to do Kerberos from the admin site and from the moss sites themselves.  So how could I set this up to have different SPNs when it was the same machine.  Well I tried setting up SPN's with url:port but that doesn't work.  The answer was host headers for the different sites.  So I setup sites like http://testsite.soapitstop.com and http://reports.soapitstop.com and then my SPNs were configured as that.  That all worked fine ... the next issue was delegation.

 Delegation breaks down into two buckets, constrained and unconstrained.  Once you have setup an SPN (and that was the first head scratcher becasue if your service account doesn't have an SPN set you will not have the delegation tab in Active Directory) you'll be able to setup delegation for that service account or machine.  The option to delegate to any service is the simplest approach and the one I elected to use for my case.  If there was some concern about the service accounts ability to delegate to more secure servers you may move to constrained delegation but there is obviously manual maintenane there beause the constrained delegation requires a list be maintained in AD of services the account is allowed to delegate too.

Now the good news was I was able to get kerberos working end to end for my fully connected to the domain case in my lab.  I could see the tickets and all the security log entries I expected on all the machines so it was good .. Right?  Nope ... Wrong.  That is unfortunately not the end of the story (did I mention I was battling Kerberos configuration for close to a month ).  The next thing was to bring in my disconnected XP workstation and see if I could authenticate to the MOSS site and start a kerberos secure ticketed session.  This was the real issue we had been trying to solve.  Initially it all blew up in my face immediately and I couldn't get anything to work.  All the logs kept saying anonymous attempts to connect were being made.  Turns out there are some security settings in IE that tell you to use Integrated Windows Authentication (Tools > Internet Options > Advances > Enable Integrated Windows Authentication). 

That seemed to do the trick in my lab but we still had some other environment issue that was causing the Kerberos traffic to not even make it to the MOSS server and I couldn't reproduce this in my lab.  After pulling in other Kerberos guru's we discovered that there is a really big difference when logging in from a non-domain joined machine.  The issue is the domain name and whether or not you are using full qualified or NetBios with your login.  If you're like me you tend to login to any site that prompts you with domain\user and then password but that will not work if you are not on the domain because the machine can not find the netbios name without being joined to the domain.  so you have to do fqdn\user to make this work.  Ugh....all of these many configuration changes, logs, and re-deployment ideas and it ends up being a problem with the way the users are supplying their credentials at login.  The big lesson learned here was that this would have been much more obvious if a network capture was taken from the client machine and the server as there were definately DNS misses in the client log that forced it to never make the connection over to the server.

One other interesting idea that came up during all of this debugging was the idea of using Protocol Transition to take a users credentials supplied via basic authentication and startup a kerberos session without even supplying a password.  I actually didn't even realize this was possible so I wanted to mention it here.  It does come with some setup requirements,  specifically,  you have to use constained delegation because you don't want the protocol transition to allow any user to be able to do anything to any machine/service in your entire domain.  Protocol transition also requires code to work so if you were going to set this up in MOSS you'd have to write a custom auth provider and plug it in.  If you combined this with the security zone configuration referenced earlier you would have a mechanism for doing kerberos authentication from both the intranet and the extranet scenario.  Typically this is done with a seperate user store but this might be an interesting option.  I didn't take it far enough to give a list pros, cons, or gotchas but in theory it sounded very interesting.

In the end,  this did end up being purely an infrastructure and user training issue but along the way I looked at pretty much every Kerberos configuration setting and logging/tracing option available.  If you're having headaches getting Kerberos setup just step back and take a deep breath and look at some of these diagnostics options or references below and you'll get through it. 

Other Resources:

Kerberos Explained: (http://technet.microsoft.com/en-us/library/bb742516.aspx) - This link is a little old and has some information in it that mislead me initially.  Specifically,  there is a statement that sort of implied a non domain joined machine would automatically fail over to NTLM.  That is in fact not the case and Kerberos tickets can be managed from any machine that can make a connection to the KDC.

Protocol Transition: (http://msdn.microsoft.com/en-us/library/ms998355.aspx) - As I said above,  I didn't get to the point where I got this all working but it is a very interesting option for starting a Kerberos session with any valid user name.

How IIS authenticates Browser Clients: (http://support.microsoft.com/default.aspx?scid=kb;EN-US;264921) - This one is a good short explanation of the different ways a browser will be authenticated using the various protocols and the pre-requisites for each.

How Kerberos V5 Works: (http://technet.microsoft.com/en-us/library/cc772815.aspx ) - This really had all the information I eventually needed but it's a super long write up and much of what you might look at is distracting.  The best detail for this scenario that I got from here was "To use Kerberos authentication in a remote untrusted domain, the user must supply credentials as part of the connection request. Otherwise, the KRB_TGS_REQ message fails, and the system tries the next security provider, typically NTLM.”  This brought my focus back to IE and the user after weeks of picking through SPNs logs and all sorts of other configuraiton items for Kerberos.