Kernel debugging with Qemu and WinDbg

If you're used WinDbg before, you might already know that you can debug the whole Windows operating system with it. To do that, you must have two Windows operating systems, where the first one is the one we'll be debugging and the second is where we'll be debugging from. The very first thing that we need to do is install WinDbg on both operating systems.

Usually I download the complete Windows Driver Kit from http://www.microsoft.com/en-us/download/details.aspx?id=11800 as can be seen on the picture below.

After downloading and installing WDK, the WinDbg will be located in the C:WinDDK folder as seen on the picture below.

At this point we can start windbg.exe and pin it to the taskbar, so we can easily start/stop it. This must be done on the host virtual machine, where we'll be debugging from.

On the guest virtual machine, we have to change the boot.ini to change the boot options, which are needed if we would like to debug the operating system. We have to run the following commands in the cmd.exe, which has to be started as Administrator to give us enough permissions.

[plain]

> bcdedit /set debug on

> bcdedit /set debugtype serial

> bcdedit /set debugport 1

> bcdedit /set baudrate 115200

> bcdedit /set {bootmgr} displaybootmenu yes

> bcdedit /timeout 10

[/plain]

After restarting Windows, the boot menu when starting Windows will look like this; notice the added "[debugger enabled]" string, which indicates that we'll start Windows under the previously entered debugging commands.

Qemu, character devices and serial ports

A character device on Linux is a special file though which we can send data to devices one character at a time, and is often used for streaming the data communication between the operating system and the hardware device. In Qemu we can create a character device by using the -chardev option, which accepts a number of backends described below. Note that each character device must have an id, which uniquely identifies the device. The options below are summarized after [1].

Backend Description

null A device that drops any data it receives and doesn't send any data.

socket Creates a TCP or Unix socket, which we can use for bidirectional communication.

udp Creates a UDP socket, which we can use to send traffic over.

msmouse Qemu's emulated msmouse events are forwarded to the guest.

vc Is used to connect to the qemu text console.

file Creates a character device, which writes all the received data from a guest to a file.

pipe Creates a pipe, which we can use for bidirectional communication with the guest.

console Sends traffic from guest to qemu's standard output and is only available on Windows hosts.

serial Creates a serial device to which the guest can send data.

pty Creates a new pseudo terminal on the host.

stdio Connects to stdin/stdout of the qemu process.

braille Connects to local BrlAPI server.

tty Connects to local tty device.

parport Connects to local parallel port.

spicevmc Connect to the spice virtual machine channel.

We can achieve what we want by creating a serial port with qemu, which can be done by using a "-serial dev" option with qemu, where the dev specifies the character device, which can be one of the devices listed in the table below [1].

Option Description

-serial vc[:WxH] Virtual console with optional width and height parameter.

-serial pty A new pseudo TTY will be allocated.

-serial none No device is allocated.

-serial null A void device.

-serial file:filename A new file is created, where the output is written.

-serial stdio Standard input/output.

-serial pipe:filename A new file is created acting as a pipe.

-serial unix:path A unix socket is used.

Remember that we can use either -chardev, which also needs the accompanying -device isa-serial part, or -serial, which provides both the frontend (host) and the backend (guest) part of the connection. The -serial is therefore there only for convenience, so we don't have to use two parameters, but only one. Those options apply when we would like to run both VMs on the same host computer, but remember that we can also run the two VMs on different host computers, but at that time we must use sockets to exchange WinDbg messages.

Kernel debugging on the same host

Let's first take a look at how we can debug the Windows kernel when both VMs are running on the same host.

There are two ways that we can enable kernel debugging. The first is by using the -chardev/-device options, while the shortcut is by using the -serial option. On the first virtual machine we can use one of the following two options to enable the server side of the communication: the debugger.

[plain]

-chardev socket,id=serial0,path=/tmp/debugpipe,server,nowait

-device isa-serial,chardev=serial0,id=serial0

[/plain]

or

[plain]

-serial tcp::9090,server,nowait

[/plain]

In the second VM, we must enable the appropriate options based on the options set in the first VM. We also have two options that are very similar to what we've used with the first VM, but are slightly different. On the second virtual machine we can use one of the following two options to enable the client side of the communication: the debuggee.

[plain]

-chardev socket,id=serial0,path=/tmp/debugpipe

-device isa-serial,chardev=serial0,id=serial0

[/plain]

or

[plain]

-serial tcp:127.0.0.1:9090

[/plain]

To start kernel debugging, we need to press the File – Kernel Debug in Windbg (in the first VM of course), and set the baudrate/port (the defaults are fine in our case). After pressing OK, the WinDbg will be waiting for the debuggee to connect. To make that happen, we need to start the second VM, which will connect to the serial port and our WinDbg (in first VM) will be able to catch it. When that happens, we can break the execution of the Windows operating system, which will enable us to enter commands at the "kd>" prompt.

At that point we won't be able to use the debugged Windows VM, because the execution of it is stopped. To run the debuggee again and let us interact with it, we must run it with the "g" command.

Kernel debugging on different hosts

When we would like to run the two Windows machines on different host computers, we can easily do so by using the serial console based on TCP. This can be done by using the "-serial tcp" command-line option of the Qemu. The first host virtual machine must create the listening port by using the option below:

[plain]

-serial tcp::9090,server,nowait

[/plain]

This will start listening on port 9090, which we can verify with the netstat command as seen below.

[plain]

# netstat -luntp

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name

tcp 0 0 0.0.0.0:9090 0.0.0.0:* LISTEN 32745/qemu-system-x

[/plain]

The guest virtual machine must then connect to that port with the following command-line option. In this case, we'll be connecting to the socket on the same computer, which is why we'll use the 127.0.0.1 IP address, but if you're on a different computer you just have to change the IP address.

[plain]

-serial tcp:127.0.0.1:9090

[/plain]

At first we must start host virtual machine, which will create the listening socket. After logging into the Windows operating system, we need to start WinDbg and go to File – Kernel Debug, where we can edit the options of debugging over the COM port. On the picture below, we can see the default options, which are already correctly set and correspond with our previously entered bcdedit commands entered in Windows Guest virtual machine.

At that point, we must merely press the OK button, which will open the COM1 port and will wait for the Windows Guest virtual machine to connect. This can be seen on the picture below.

After that we must start the Windows Guest virtual machine, which will by default try to connect to the COM1 serial port, because we edited the only booting option with bcdedit. Once the Guest virtual machine connects to the serial port, the following will be seen in the WinDbg started in the host virtual machine. Notice that the Windows operating system is connected to WinDbg and stopped, and it's our job to enter the g command to run the operating system, at which point the booting process will be able to continue.

On the picture above we can notice that the symbol file was not found. Whenever using WinDbg for debugging I always set the _NT_SYMBOL_PATH environment variable to the "srv*C:symbols*http://msdl.microsoft.com/download/symbols" value as seen below.

After applying the changes, the WinDbg will automatically pick up the changes and download the symbols from Microsoft symbol store and cache them in C:symbols directory.

Conclusion

In this article we presented how we can debug the Windows kernel under Qemu by using two virtual machines. First we installed the debugging tools in first virtual machine and enabled the debugging options in boot.ini in the second virtual machine. From there on, we only need to set the right parameters to the qemu-kvm command.

We've looked at the -chardev/-device and -serial parameters that can be used when kernel debugging with WinDbg. I recommend using the -serial parameter, which enables us to communicate over the socket, which is useful when we would like to run the debugger and debuggee machines on different hosts. By using this option, we can merely change the IP address and everything works as before.

Become a certified reverse engineer!

Get live, hands-on malware analysis training from anywhere, and become a Certified Reverse Engineering Analyst.

Start Learning

Sources

• Qemu man page
• Qemu, Windows guest debugging