Windows device drivers: Notes to self

Yet another messy post with just things I wrote down while developing a device driver for Windows. Not necessary coherent, not necessarily accurate.

General notes while playing around

• To start Windows 10 allowing installation of unsigned drivers, hold Shift while selecting “Restart” in the Power menu. Windows will appear to restart normally, but enters a menu before shutting down. Go Troubleshoot > Startup Settings The Windows 7-like warning appears during installation, and the driver installation survives the reboot it regular mode. However text saying “Test Mode” appears at the bottom right of the desktop on following boots. Windows may then randomly refuse to use the driver on the basis that it isn’t signed on following reboots.
• It seems like Windows restarts immediately after installing a driver if the installation fails. Message in event log: “The process C:\WINDOWS\system32\mmc.exe (DESKTOP-xxxxx) has initiated the restart of computer DESKTOP-xxxxx on behalf of user DESKTOP-xxxxx\theuser for the following reason: Hardware: Installation (Planned)
• “Read the source”, WDK-style: Read the inc\ddk\wdm.h include file, which supplies a lot of information and hints. Sometimes it’s much better than the docs. There are also sources for the implementation of basic functions such as open(), printf() etc. in Visual Studio’s installation directory.
• A list of status codes can be found here. Possibly relevant in particular: STATUS_UNSUCCESSFUL,  STATUS_NOT_IMPLEMENTED, STATUS_INVALID_PARAMETER, STATUS_NO_MEMORY, STATUS_INVALID_DEVICE_REQUEST, STATUS_DEVICE_BUSY, STATUS_END_OF_FILE, STATUS_ACCESS_DENIED, STATUS_DATA_ERROR, STATUS_NO_SUCH_DEVICE, STATUS_NO_SUCH_FILE, STATUS_OBJECT_NAME_NOT_FOUND,  STATUS_INSUFFICIENT_RESOURCES, STATUS_IO_TIMEOUT, STATUS_FILE_FORCED_CLOSED,  STATUS_CANCELLED, STATUS_NOT_FOUND, STATUS_RETRY, STATUS_RECEIVE_PARTIAL
• IRP’s major determine the dispatch routine launched (as set by DriverEntry). The minor is interpreted by the dispatch routine itself.
• A list of safe string functions: In MSDN’s page
• How to use symbolic links to create a name space: here.
• To see the internal object structure (and symbolic links), download WinObj from somewhere.
• Sniff IRPs in the system: Download IrpTracker.
• MSDN’s explanation on resource rebalancing, why, how and what to do about it.
• On a typical open for write, DesiredAccess=0x00120196 (FILE_GENERIC_WRITE | READ_CONTROL). On an open for read, DesiredAccess=0x00120089 (FILE_GENERIC_READ | READ_CONTROL). This is based upon command line > and <. Trying to “dir” the file gives DesiredAccess=0x00000080.
• The difference between IRP_MJ_CLEANUP and IRP_MJ_CLOSE: IRP_MJ_CLEANUP is called when the file is officially closed (all owners of the file handle have closed it). IRP_MJ_CLOSE is called when the file handle can be removed, which is when all outstanding IRPs have been completed as well. Note that IRP_MJ_CLEANUP dispatch routine must walk through the queue and cancel all requests (Why doesn’t the system do this?). Note that non-I/O IRPs for the file handler may still arrive for the file handler.
• Do implement a handler for IRP_MJ_SET_INFORMATION, so that such an IRP doesn’t fail. Even a yes-yes handler doing nothing but returning success. In particular, Cygwin sends this IRP with FileEndOfFileInformation when writing to a regular file. Go figure.
• Using MmGetMdlByteCount(irp->MdlAddress) to get the number of bytes to read or write is bad: It causes a bugcheck if the number of bytes to read or write is zero. Which is bad behavior from the application’s side, and still.
• Not directly relevant, but user space applications with POSIX functions such as open(), read() close() and friends should include <io.h> rather than <unistd.h> which doesn’t exist on VC++.
• Setting the DeviceType parameter is utterly important, since the native fopen() and open() calls will fail with an “Invalid Argument” error otherwise. The parameter in the call to IoCreateDevice() is ignored since the actual type is taken from the PDO. Hence a line in the INF file for setting up a registry value is necessary.
• When seeking a file, be either with POSIX-style _seek() and friends, or with native SetFilePointer(), this merely updates CurrentByteOffset, but no IRP is issued, so the device has no idea it happened at all. In particular, no IRP_MJ_SET_INFORMATION IRP with class FilePositionInformation is issued, despite what one could expect. At least so it works in Windows 7.
• pnpdtest (which is part of the WDK, and tge wtllog.dll is also somewhere in its directories) should be run with the verifier (standard Windows utility) configured for toughest tests specifically on the driver. Just type “verifier” at command prompt, reboot, and run pnpdtest. To run verifier with a good set of tests, go
  > verifier /volatile /flags 0xbfb /adddriver mydriver.sys
• Unlike Linux, the allocation of PCI resources (BAR addresses and interrupts) is not exclusive. As a result, if a device driver doesn’t deallocate the interrupt resource at unload, and hence the pointer to the ISR remains, attempting to reload the driver will cause a jump to the unloaded ISR pointer and a bugcheck. In Linux this wouldn’t happen, because the newly loaded instance of the driver wouldn’t pass the stage of getting the resources.
• devcon is the old tool for installing and looking at driver information, with source code in the Windows Driver Samples git repo. pnputil should be used instead, according to Microsoft.

Runlevels (IRQLs)

A very short summary (see Microsoft’s page for more):

• PASSIVE_LEVEL: The level of user space applications, system threads and work items specially generated to run in this level. May block, but in most cases it’s not allowed because of arbitrary thread context.
• APC_LEVEL: The special level for delivering data to user applications. Rarely seen, but often considered.
• DISPATCH_LEVEL: Used in DPC (Deferred Procedure calls) queued by ISR or by a custom call. Also, when acquiring a spinlock, the runlevel is raised to DISPATCH_LEVEL.
• Higher levels: When serving interrupts. No kind of mutex can be acquired (neither spinlocks). Some simple memory operations and queuing DPCs is more or less what’s left to do.

Notes:

• Paged code runs below DISPATCH_LEVEL. So when PAGED_CODE appears in the beginning of a function, it’s at most at APC_LEVEL.
• DPCs run at DISPATCH_LEVEL. When requested, a single instance is queued and executed when possible. The DPC instance is dequeued just before execution, so if another DPC request occurs immediately after launching a DPC, another run will take place, possibly simultaneously on another CPU. On the other hand, if several requests are made before execution is possible, the DPC is run only once on behalf of these requests.
• Custom DPCs can be queued from any runlevel. If the “background routine” needs to be queued from DISPATCH_LEVEL and down, use a work item instead.
• Confusingly enough, most (almost all) IRP dispatch calls are made in passive level. To make things more difficult, read, write and ioctl IRPs can go up to DISPATCH_LEVEL.

Bug check handling

What to do when the Blue Screen (BSOD) pays a visit. Or more precisely, how to catch exactly where your code wanted to use a zero pointer or something.

It’s important to save the entire binaries directory, including .obj files, for the driver distributed, since those files are necessary for bug check dissection.

The files are typically something like:
C:\Windows\Minidump\030312-17503-01.dmp
C:\Users\theuser\AppData\Local\Temp\WER-33571-0.sysdata.xml

However the XML file isn’t necessary for the bugcheck analysys, only the dump file.

To enable generation of a dump file, go to “System” from the Start menu, pick “Advanced system settings”  and click “Settings…” in the “Startup and Recovery” section. In the “System Failure” section, select “Small memory dump (128 kB)” in the drop-down menu. This option is possible only if there is swap space enabled. An alternative directory for the dump file can be selected there as well (but why?).

Analyze a bugcheck dump (change last file, of course). Be patient. Each step takes time.

> C:\WinDDK\7600.16385.1\Debuggers\kd.exe -n -y srv*c:\symbols*http://msdl.microsoft.com/download/symbols -i C:\devel\objchk_win7_x86\i386 -z C:\copies\022512-24414-01.dmp

The symbols which are downloaded are stored in C:\symbols (as required). These relate to a given version of the Windows kernel (and loaded drivers), so keep a different directory for each platform (?), or delete the directory altogether before invoking the debugger if unsure. It’s a cache, after all.

When those downloads are finished, go

0: kd> !analyze -v

Quit with “q”

To get a disassembly of the relevant code, go something like:

1: kd> u nt!PnpCallDriverEntry nt!PnpCallDriverEntry+0x4c

This prints out the disassembly from the beginning of PnpCallDriverEntry() to offset 0x4c. If the latter offset is the one appearing in the stack trace, the disassembly goes until the command after the call to the function above in the trace (because the address in the stack is the one to return to, not the caller’s address). The interesting disassemblies are of course those on the driver itself, not the system code as shown in this example.

On a 32-bit architecture, it’s also possible to disassemble the object file (but why?). Linux’ objdump -d works on Windows’ object files, surprisingly enough, and then there’s Microsoft’s counterpart:

> dumpbin.exe /disasm \path\to\objchk_win7_x86\i386\project.obj > project.asm

Note that we disassemble the object file to which the code belongs. That alone is a good reason to maintain an exact snapshot of released versions. The .sys file can be disassembled to verify that nothing has changed, but that disassembly doesn’t contain symbols names.

Unfortunately, the build for 64 bit (amd64) creates .obj files that can’t be disassembled, so only the .sys files can be used, and once again, they don’t contain symbols.

dumpbin.exe is part of the Visual C++ package. The necessary file bunlde are in my resumption directory, containing dumpbin.exe, link.exe, msdis140.dll and mspdb71.dll

---

April 2021 update: After another round of work on Windows drivers (still WDM), I’ve added a few more notes:

Another look at IRP handling

While the do’s and don’ts of IRP processing with the WDM model are well-documented, somehow I’ve never seen the idea behind the API explained. And it’s a fascinating one (not clear what kind of fascinating, though): It mimics a subroutine call, and implements the call stack as an array going with the IRP.

This is easiest explained by comparing with Linux’ driver API: For example, a read() system calls ends up as a call to the related function presented by the driver. That driver may call other functions in the kernel to fulfill its request. It may sleep waiting for data, causing the user-space process to sleep. It may call functions belonging to drivers at a lower level, i.e. closer to the hardware. Once it’s done, it returns, and the user space program gets its data.

Microsoft took an asynchronous approach, meaning that there is no direct connection between the user space program that initiated the read() call and the threads in the kernel that handles it. Rather, a data structure (the IRP) containing details on the read() request is set up, and then the execution flow goes event-driven style. Sometimes, the IRP is fulfilled by a single call to a handler functions, but in general, it’s juggled among several such. The fulfillment is cut into fragments, each running in a function of its own, and each needing to discover the overall state of the progress, not to mention tackle race conditions with the possibility that the IRP is canceled.

But the interesting point is the stack. And it has two meanings, actually: One is that drivers are stacked up, so there’s a low-level driver (say, the USB hub driver, which processes URBs) and higher-level drivers (e.g. for a sound card). Obviously, the intended usage was that if the user-space process makes a read() call for X bytes of data, an IRP is set up for that request, and handed over to the sound card’s driver for fetching X bytes of sound data. This driver goes on by setting up a request which is tied to the same IRP, for a data read URB of X bytes, and passing the IRP to the driver below it. And so it goes down the stack until some driver does the actual I/O. So the IRP represents the task to be fulfilled, and each driver in the stack presents its interpretation of the request.

And this brings me to the second meaning of “stack”: An execution stack. If the same thing was to run Linux kernel style, this would take the form of the upper driver literally calling a function belonging to a lower level driver, which could go on deeper and deeper until the ultimate driver is reached.

Apparently, Microsoft decided to mimic this execution stack by attaching an array of IO_STACK_LOCATION structures to the IRP’s main data structure, which is, well, the IRP stack . One way to look at these structures is that they contain the arguments of the function call to the driver beneath. So each of these IO_STACK_LOCATION structs contain the stuff that would have been pushed into the stack, had this lower-driver call been done Linux style. Which would have been the case, had Microsoft decided to give each IRP an execution thread that could sleep.

But wait, you say. There’s a return address too in the stack as well when a function is called. Well, I’m just getting to that.

So the idea is like this: The handling of an IRP consists of three parts: Things to do before calling the driver beneath, calling the driver beneath, and things doing after that. In Linux style it would have been something rhyming with

int stuff_handler( ... arguments ...)
{
   do_some_stuff();

   call_lower_driver( ... arguments ... );

   do_stuff_afterwards();
}

But this doesn’t work with Window’s model, because in most interesting cases, the call to the lower driver can be pended. The do_stuff_afterwards() needs to be done after the lower drivers have finished their business. That’s where the completion routine comes in: It contains the code for do_stuff_afterwards().

So effectively, the call to IoSetCompletionRoutine() takes the role of pushing the address to return to into the execution stack, in a regular function call. The completion routine is sort-of the return-from-function address for the IoCallDriver() call.

The completion routine can return one of two statuses:

• STATUS_CONTINUE_COMPLETION, which has the effect of returning from stuff_handler(), i.e. giving control to the function that called stuff_handler(). In Window’s stack model, this results in calling the completer function of some driver above. That is, the do-stuff-afterwards part of the upper driver.
• STATUS_MORE_PROCESSING_REQUIRED, meaning nope, don’t return from stuff_handler. Because all execution is done with functions, the completion function has to return when there’s no more useful things to do, but in this game of faking an execution thread, this status means “let’s pretend I never returned”.

So finally, we have IoCompleteRequest(). It simply means “stuff_handler returns now”. It’s the opposite to STATUS_MORE_PROCESSING_REQUIRED. It’s not (necessarily) called from a completer function, but it has the effect of returning from it with a STATUS_CONTINUE_COMPLETION.

The completion routine doesn’t have to be set. This is like finishing a Linux-style handler with

  return call_lower_driver( ... arguments ... );

which is typically translated into a JMP.

Looking at the IRP framework as a mimic of an execution thread is quite counterintuitive, in particular the idea that returning from that faked function is done by calling a function in the actual C code. Nevertheless, it helps somewhat when trying to make sense of some of the whole IRP framework.

Disassembling amd64 files

At some point, I wanted to disassemble the obj files created in an amd64 compilation. Dumpbin gave me:

Microsoft (R) COFF/PE Dumper Version 7.10.3077
Copyright (C) Microsoft Corporation.  All rights reserved.

Dump of file thefile.obj

FileType: ANONYMOUS OBJECT

and indeed, it’s not a COFF file, but rather an Anonymous Object file, with a rather odd magic word:

00000000  00 00 ff ff 01 00 64 86  a5 68 78 60 38 fe b3 0c  |......d..hx`8...|
00000010  a5 d9 ab 4d ac 9b d6 b6  22 26 53 c2 9b 81 01 00  |...M...."&S.....|
00000020  13 0c 07 00 a5 68 78 60  89 80 01 00 0f 00 00 00  |.....hx`........|
00000030  00 00 00 00 2e 64 72 65  63 74 76 65 00 00 00 00  |.....drectve....|

After the first four bytes, there a version number (0x0001) and then an 0x8664 signature for amd64. This is most likely a result of compiling with the /GL option for a whole program optimization. Hence there is nothing to disassemble here: Unlike a COFF file, which is just relocated into the final output, it appears like the content of this kind of object file is up for some additional mangling. Hence a disassembly of this file would have been useless, if at all possible.

So bottom line: Don’t even try. The .sys file can be disassembled, but it’s without symbols. So the sane way seems to be to use the kd debugger when disassembly is needed.

Bugcheck for memory leak

This is what it looks like at !analyze -v (minus lots of mumbo-jumbo)

Loading unloaded module list
.................
3: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

DRIVER_VERIFIER_DETECTED_VIOLATION (c4)
A device driver attempting to corrupt the system has been caught.  This is
because the driver was specified in the registry as being suspect (by the
administrator) and the kernel has enabled substantial checking of this driver.
If the driver attempts to corrupt the system, bugchecks 0xC4, 0xC1 and 0xA will
be among the most commonly seen crashes.
Arguments:
Arg1: 00000062, A driver has forgotten to free its pool allocations prior to unloading.
Arg2: 84caebd4, name of the driver having the issue.
Arg3: 84b9b418, verifier internal structure with driver information.
Arg4: 00000001, total # of (paged+nonpaged) allocations that weren't freed.
        Type !verifier 3 drivername.sys for info on the allocations
        that were leaked that caused the bugcheck.

[ ... ]

BUGCHECK_STR:  0xc4_62

IMAGE_NAME:  thedriver.sys

DEBUG_FLR_IMAGE_TIMESTAMP:  60635055

MODULE_NAME: thedriver

FAULTING_MODULE: 99a9b000 thedriver

VERIFIER_DRIVER_ENTRY: dt nt!_MI_VERIFIER_DRIVER_ENTRY ffffffff84b9b418
Symbol nt!_MI_VERIFIER_DRIVER_ENTRY not found.

CUSTOMER_CRASH_COUNT:  1

DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT

PROCESS_NAME:  System

CURRENT_IRQL:  2

LAST_CONTROL_TRANSFER:  from 82d51f03 to 82af9d10

STACK_TEXT:
8afa8a28 82d51f03 000000c4 00000062 84caebd4 nt!KeBugCheckEx+0x1e
8afa8a48 82d565eb 84caebd4 84b9b418 99a9b000 nt!VerifierBugCheckIfAppropriate+0x30
8afa8a58 82a29e8a 84caeb78 99a9b000 40000000 nt!VfPoolCheckForLeaks+0x33
8afa8a94 82bae69f 84caeb78 99a9b000 40000000 nt!VfTargetDriversRemove+0x66
8afa8aa8 82bae338 82b657e0 849a4a70 00000000 nt!VfDriverUnloadImage+0x5e
8afa8ae0 82baf58d 84caeb78 ffffffff 00000000 nt!MiUnloadSystemImage+0x1c6
8afa8b04 82cd8517 84caeb78 849c09c8 84c59668 nt!MmUnloadSystemImage+0x36
8afa8b1c 82c3e6f4 84c59680 84c59680 84c59668 nt!IopDeleteDriver+0x38
8afa8b34 82a85f60 00000000 84ab9818 84ab9768 nt!ObpRemoveObjectRoutine+0x59
8afa8b48 82a85ed0 84c59680 82bd6cb2 849c4838 nt!ObfDereferenceObjectWithTag+0x88

8afa8b50 82bd6cb2 849c4838 84ab9750 84ab9768 nt!ObfDereferenceObject+0xd

[ ... ]

STACK_COMMAND:  kb

FOLLOWUP_NAME:  MachineOwner

FAILURE_BUCKET_ID:  0xc4_62_LEAKED_POOL_IMAGE_thedriver.sys

BUCKET_ID:  0xc4_62_LEAKED_POOL_IMAGE_thedriver.sys

Followup: MachineOwner
---------

Just in case I’ll see one of these one day.