11 May, 2017

Using Threads Snapshot tool as postmortem debugger

This article will show you how to register and use Threads Snapshot tool to capture call stacks of unhandled exception crash of any process.

When application throws exception which is not handled by application's code (so called "unhandled exception") - a system will attach external "debugger" to application (called "Windows Error Reporting", WER). External debugger can collect information about crash. Default WER debugger will collect basic information as well as possible process dump, which can be send to Microsoft servers for analyzing.


There is a feature which allows any 3rd party to register its own external debugger, which can be used to diagnose unhandled exceptions. It is documented in the MSDN here, and in the TechNet here. Basically, you need to write a command line to invoke your debugger in HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug\Debugger registry key (it is HKLM\Software\Wow6432Node\Microsoft\Windows NT\CurrentVersion\AeDebug\Debugger for 32-bit on 64-bit). Once debugger is registered, an additional button will be available in default WER dialog:


Clicking on the "Debug" button will run registered debugger, passing information about crash. You can use such debugger to analyze the crash or collect custom information about the crash. Such debugger is called postmortem debugger or Just-In-Time debugger (JIT-debugger).

Here is a list of tools, which you can use as postmortem debuggers:
  • IDE's debugger, such as Delphi or Visual Studio.
    In fact, these IDEs will automatically register itself as postmortem debugger when you install IDE. For example, a command-line for Delphi looks like "C:\Path-To-Delphi\Bin\bordbg70.exe" -aeargs %ld %ld or "C:\Path-To-Delphi\Bin\BDS.exe" /attach:%ld;%ld - depending on your IDE version;
  • Microsoft debuggers and tools (such as WinDbg, CDB, and NTSD);
  • ProcDump tool from SysInternals;
  • Threads Snapshot tool from Neos Eureka s.a.s.

When you are working on developer's machine - you probably want a full debugger to be registered as postmortem debugger. Because once launched - it will allow you to analyze call stacks, variables, memory, etc. However, if you need to analyze crash on any other machine, then IDE debugger may not be a best option. Because you need to install IDE (or remote debugger), connect to remote machine via some sort of remote controlling software, coordinate your time with user on remote machine, etc. That is where "capture" tools come in.

(Sadly, RAD Studio IDE does not support passing JIT_DEBUG_INFO structure via command-line - which makes analyzing crash a lot more harder.)

You can install free EurekaLog tools pack on remote machine and register Threads Snapshot as postmortem debugger. You simply have to run Threads Snapshot with "/install" command line option under administrator account:


Once registered, you can click "Debug" button in WER crash dialog to capture call stacks of crashed process:


Threads Snapshot tool will create a usual .el bug report, which can be opened in EurekaLog Viewer:



In the example on the screenshots above: the unhandled exception was raised by thread function which was started via CreateThread function (it does not have any buildin try/except blocks, so any raised unhandled exception will be catched by OS).

(Note: EurekaLog-enabled applications usually handle "unhandled" exceptions from CreateThread-threads - that is, unless you disable low-level hooks or disable EurekaLog in background threads. This case is provided only as simple example.)

Obviosly, the target process should have some sort of debug information - otherwise, Threads Snapshot tool would not be able to get human-readable names for call stacks. Threads Snapshot tool support the usual set of EurekaLog's supported debug formats:

You can restore postmortem debugger registration by running Threads Snapshot tool with "/uninstall" command line option (administrator's priviledges are required).
Tags: , , , , , ,