Kernel Debugging
WinDbg Host and Target
in Virtual Machines

VMWare Setup for Kernel Debugging between Virtual Machines

VMWare provides examples that illustrate how to use virtual serial ports for debugging at the URL:

Driver Debugging with WinDbg and VMWare

I have been using VMWare since V3.0 mostly for installation debugging and occasionally for debugging an NDIS driver with WinDbg running on the VMWare host. However, I recently encountered a project that was a perfect candidate for the situation where the kernel code to be debugged is in one virtual machine (called the target virtual machine) and WinDbg (and the driver Build environment) is in another virtual machine (called the debugger virtual machine). The key points that distinguish the situation were:

bulletThe customer insisted on using an unreleased WDK version (then adopted a later WDK that I was not ready to support...).
bulletThe customer's application development environment included global Visual Studio settings that were inappropriate for general development.
bulletThe project included an NDIS 5 and a NDIS 6 protocol driver that must both interface to a common user-mode application.

To work on this project three virtual machines would be needed:

bulletDebugger Virtual Machine - A Windows XP SP3 VM with all build tools and sources installed.
bulletTarget Virtual Machine No. 1 - A Windows XP SP3 VM with all build tools and sources installed.
bulletTarget Virtual Machine No. 2 - A Windows 7 RTM VM with all build tools and sources installed.

This should be easy to setup.

The VMWare instructions include an example where you have kernel code to debug in a virtual machine (called the target virtual machine) and are running Debugging Tools for Windows (WinDbg) or Kernel Debugger (KD) in another virtual machine (called the debugger virtual machine) on the same host. These are pretty good instructions, but I am writing the remainder of this article as a "Note to Self with Pictures" for my own benefit. Maybe it will be of use to readers.

 

Building Target Virtual Machine No. 1 (WIndows XP)

I have a library of Windows virtual machines. Some in their "clean install" state and others extended to have the WDK and various Visual Studio versions installed.

To make Target Virtual Machine No. 1 I started by making a full clone of a library Windows XP SP3 VM that had Visual Studio 2008 installed. I ran this machine and configured it with the customer's desired WDK version. I used Tortoise SVN to checkout the customer's sources. I made a test build of the applications and driver to insure that the development environment was OK, but stopped short of actually installing the project output or the NDIS 5 protocol driver to be debugged.

Install WInDbg if it isn't already installed in the library VM.

The Windows XP boot.ini file must be edited exactly as it would be for any other WinDbg target using a serial port:

[boot loader]
timeout=30
default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS
[operating systems]
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Microsoft Windows XP Professional" /noexecute=optin /fastdetect
multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows XP Professional (Debug COM1)" /noexecute=optin /fastdetect /debug /debugport=COM1 /baudrate=115200

Shutdown the VM and then edit the Virtual Machine Settings Serial Port settings as you would for an ordinary VMWare debug target machine:

Now take a snapshot of Target Virtual Machine No. 1.

 

Building the Debugger Virtual Machine

Very easy to do. Target Virtual Machine No. 1 is already configured with all the software that is needed - including the customer's applications and driver sources.

Just make a clone of Target Virtual Machine No. 1. Then run it.

At a minimum you must change the Computer Name of the Debugger Virtual Machine so it won't conflict with Target Virtual Machine No. 1. You might also want to add a few features such as WinDbg map files to facilitate driver testing.

Shutdown the VM.  Then edit the Virtual Machine Settings.  The Serial Port settings are a little different as illustrated below.

Note that on the Debugger Virtual Machine I added two serial ports. The first (COM1) uses \\.\pipe\com_1 and will be used as the WinDbg serial port to debug Target Virtual Machine No. 1 (the Windows XP target). The second (COM2) uses \\.\pipe\com_2 and will be used as teh WinDbg serial port to debug Target Virtual Machine No. 2 (the Windows 7 target).

Now take a snapshot of the Debugger Virtual Machine.

 

Building Target Virtual Machine No. 2

To make Target Virtual Machine No. 2 I started by making a full clone of a library Windows 7 RTM VM that had Visual Studio 2008 installed. I ran this machine and configured it with the customer's desired WDK version. I used Tortoise SVN to checkout the customer's sources. I made a test build of the applications and driver to insure that the development environment was OK, but stopped short of actually installing the project output or the NDIS 5 protocol driver to be debugged.

Install WInDbg if it isn't already installed in the library VM.

Since this is a Windows 7 VM you must change the debug print flags and configure for serial debugging over COM1 using BCDEDIT.

Now take a snapshot of Target Virtual Machine No. 2.

 

Now for Debugging

At this point all three VMs should be ready to be used. Operation from this point on is really identical to debugging with "real iron".

Start the Debugger Virtual Machine and then WinDbg using serial ports. The WinDbg instance for debugging Target Virtual Machine No. 1 uses COM1:

 

Next start Target Virtual Machine No. 1. If you have told WinDbg to make an initial breakpoint your would see the usual command output as the target starts:

Microsoft (R) Windows Debugger Version 6.11.0001.404 X86
Copyright (c) Microsoft Corporation. All rights reserved.
Opened \\.\Com1
Waiting to reconnect...
Connected to Windows XP 2600 x86 compatible target at (Wed Sep  9 11:55:56.564 2009 (GMT-4)), ptr64 FALSE
Kernel Debugger connection established.  (Initial Breakpoint requested)
Symbol search path is: srv*\\SandBox\Locker\WebSymbols*http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows XP Kernel Version 2600 MP (1 procs) Free x86 compatible
Built by: 2600.xpsp_sp3_gdr.090206-1234
Machine Name:
Kernel base = 0x804d7000 PsLoadedModuleList = 0x8055d720
System Uptime: not available
Break instruction exception - code 80000003 (first chance)
*******************************************************************************
*                                                                             *
*   You are seeing this message because you pressed either                    *
*       CTRL+C (if you run kd.exe) or,                                        *
*       CTRL+BREAK (if you run WinDBG),                                       *
*   on your debugger machine's keyboard.                                      *
*                                                                             *
*                   THIS IS NOT A BUG OR A SYSTEM CRASH                       *
*                                                                             *
* If you did not intend to break into the debugger, press the "g" key, then   *
* press the "Enter" key now.  This message might immediately reappear.  If it *
* does, press "g" and "Enter" again.                                          *
*                                                                             *
*******************************************************************************
nt!RtlpBreakWithStatusInstruction:
8052b5ec cc              int     3
kd> g
ERROR: DavReadRegistryValues/RegQueryValueExW(4). WStatus = 127
ERROR: DavReadRegistryValues/RegQueryValueExW(5). WStatus = 127
ERROR: DavReadRegistryValues/RegQueryValueExW(6). WStatus = 127

Within the debug target you can run Visual Studio 2008 to debug the user-mode side of the project.

If your VMWare host has enough memory you can concurrently run and debug Target Virtual Machine No. 2 using a second instance of WinDbg using COM2.

 

More VMs the Merrier

The customer's applications actually communicate with each other over the network. So, having the two target VMs running concurrently is actually necessary for testing. VMWare supports this using a variety of virtual networking options. I simply used bridged networking which effectively places the target VMs on a local intranet. The bridged networking lets the target VMs communicate with each other.

At later stage of testing I may want to run the customer's driver and application and driver on real hardware; the bridged networking lets me make a hybrid test network consisting of several VMs and real hardware nodes.

 

WHQL Testing using VMWare Virtual Machines

The VMWare implementation (as of 6.5...) does not provide adequate emulation of low-power states and doesn't support RTC wakeup. This makes it impossible to run WHQL tests that involve sleep stress in a VMWare virtual machine. However, you may be able to perform other tests that don't involve using low-power states.

I would not give any serious consideration to making a WHQL submission based on testing where the DTM Client was hosted on a virtual machine.

 

Topic Status

November 3, 2009 Initial posting.

PCAUSA Home · Privacy Statement · Products · Ordering · Support · Utilities · Resources
Mailing Lists  · PCAUSA Newsletter · PCAUSA Discussion List
Rawether for Windows, Rawether .NET, WinDis 32 and NDIS Press are trademarks of Printing Communications Assoc., Inc. (PCAUSA)
Microsoft, MS, Windows, Windows Vista, Windows 95, Windows 98, Windows Millennium, Windows 2000, and Win32 are registered trademarks and Visual C++ and Windows NT are trademarks of the Microsoft Corporation.
Copyright © 1996-2010 Printing Communications Assoc., Inc. (PCAUSA)
Last modified: January 17, 2010