Troubleshooting IIS7 503 "Service unavailable" errors with startup debugging

In a previous post titled Where did my IIS7 server go? Troubleshooting "service unavailable" errors, I covered the basic steps for dealing with cases where the IIS7 web server fails to process the request.  This error mostly occurs whenever HTTP.SYS, the kernel HTTP driver that manages http connections for IIS, fails to create an IIS worker process to process the request.

This failure is typically caused by a critical error during worker process initialization, or more likely an unhandled exception / access violation occurring during worker process startup.  The most common instance of this is a module AVing inside its RegisterModule function during worker process initialization.  It can also be caused by the process going away due to an unhandled exception / AV during the processing of the request that causes the process to be torn down. 

After a certain number of failures, the application pool will trigger Rapid Fail Protection, a WAS feature designed to stop application pools with a persistent failure condition to avoid an endless loop of failing to start worker processes.  At this point, all requests to applications within the stopped application pool will result in the 503 error, and the application pool will need to be re-started manually.

My previous post covers the steps necessary to diagnose most of the error conditions that may cause 503 errors.  In most of these cases, there should be an event log event in the "Application" event log that indicates why the worker process has terminated.

However, in the case of an unhandled exception or AV, you may need to run the worker process under a debugger to get more information about the error.  This can give you critical insight into the problem, including the DLL causing the exception, and the exception stack trace, which you can then use to remove the offending module, or provide details to the developer to fix the problem.

1) First, you need to download and install the latest debugging tools.

2) Restart the application pool if it entered RFP (rapid fail protection) and is currently stopped:

> %windir%\system32\inetsrv\appcmd start apppool <APPPOOLNAME>

Where <APPPOOLNAME> is the name of your application pool.

3) If the problem is happening during request processing of a particular request, you can try to make a different request to the site that does not cause the exception, and then attach the debugger by running the following from the command-line:

> windbg -g -p <PID>

Where <PID> is the PID of the IIS worker process you are debugging.  You can get it from Task Manager or by running tasklist /fi "imagename eq w3wp.exe".

If you have the debugger attached at this point, skip steps 4 and 5

4) If the problem is happening on every request, then you won't have a worker process running that you can attach to since it will go away as soon as you make a request.  In this case, you will need to attach a debugger immediately after the worker process is started, so that you can catch a startup exception before it tears down the process.

This is where startup debugging comes into play.  It is done by configuring the debugger to attach to the process as soon as its started, by setting the Image File Execution Options registry key from the command line:

> REG ADD "HKLM\Software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\w3wp.exe" /v Debugger /d "c:\debuggers\ntsd.exe -server npipe:pipe=w3wp -g" /t REG_SZ /f

This creates a sub-key named "W3WP.EXE" in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options, and in that sub-key creates a String value named "Debugger" and set its value to "c:\debuggers\ntsd.exe -server npipe:pipe=w3wp -g".

For this to work, you have to make sure that the path to the debugger is correct.  If the debuggers are already in the path, you can omit the full path.

5) Now, lets attach.  

- Make the request that causes the crash 

This will trigger the IIS worker process to be started, and will automatically attach the debuger, breaking into the process before any IIS initialization takes place.

- Now, connect to the debugger from the command-line:

> windbg -remote npipe:server=localhost,pipe=w3wp

5) Inspect the exception stack trace

The debugger window should now be broken in at the location of the exception.  You can inspect it further by typing "k" and pressing Enter.

This should give you the full stack trace of the exception. 

If you are missing symbols, you can get them from the Microsoft's public symbol server.  To do that, type the following in the debugger:

> !sympath+ SRV*c:\symbols*http://msdl.microsoft.com/download/symbols
> .reload
> k

When you are done, be sure to disable the automatic debugging on start-up:

> REG DELETE "HKLM\Software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\w3wp.exe" /v Debugger /f

Hope this helps you get to the bottom of particularly nasty start-up crashes.  If the exception stack is within IIS or a third party module, please post it on forums.iis.net so we can take a look.

Thanks,

Mike