Exercise: booting

To compile and run lvisor, you need to set up the toolchain, as described in the tools guide.

You’ll find the Intel SDM, Vol 3 useful for exercises and projects, especially the virtualization part, Vol 3C.

Cloning lvisor

The course Git repository is on the CSE GitLab. Follow the instructions there to set up your ssh keys if you haven’t.

You need to clone the lvisor repository, by running the commands below.

$ git clone https://gitlab.cs.washington.edu/cse481a/lvisor-18wi.git lvisor
...
$ cd lvisor
$ 

You have read access to this repository, but not write. We recommend creating a fork of this repo to keep any changes you make to lvisor.

Note that lvisor is a minimal hypervisor: it supports only one guest OS. You are encouraged to extend lvisor, such as to support running Linux as a guest OS or multiple guests at the same time.

Building and running

From the root directory of lvisor, you will see the following subdirectories:

• boot: boot configuration & tools;
• firmware: code for bootstrapping a guest kernel;
• include: headers;
• kernel: common code shared by lvisor and the lv6 kernel;
• lib: utility functions for manipulating and printing strings;
• tests: guest OSes, currently xv6, lv6, and Linux;
• vmm: the main component of lvisor.

The config.def file contains the default configurations. You can either edit these directly, or create a new file named config.mk under the same directory and override these variables.

Run make to build lvisor:

$ make
...
     LD       o.x86_64/vmm.elf
     GEN      o.x86_64/vmm.bin

There are two commands to run lvisor and guest OS kernels:

• make qemu will run lvisor in QEMU, along with the kernel specified in your configuration as the guest OS. By default it will run the lv6 kernel. To run xv6 instead, try make qemu KERNEL=tests/xv6/kernelmemfs or add KERNEL=tests/xv6/kernelmemfs to your config.mk.

• make qemu-kernel will run the guest OS directly in QEMU, without using the VMM. This is sometimes useful for debugging. Similarly, this will run lv6 by default.

Running lvisor and guests

Let’s start with lv6, a minimal x86-64 OS. Browser the source code under tests/lv6/:

• head.S: set up the 64-bit mode;
• main.c: initialize the kernel and user space;
• syscall.c: the system call table;
• user.S: user space code invoking two system calls.

First, let’s start the lv6 directly without lvisor:

$ make qemu-kernel
...
[    0.000000] tsc: fast TSC calibration using PIT
[    0.073976] tsc: 4199.882 MHz
[    0.074187] booting...
[    0.074490] hey 481
[    0.074636] bye 451

You should see that lv6 prints out timestamps and two system calls. This is similar to OS kernels you have seen from CSE 451.

Now, let’s run lv6 again, but this time as a guest OS of lvisor:

$ make qemu
...
[    0.000000] tsc: fast TSC calibration using PIT
[    0.197574] tsc: 4200.018 MHz
...
[    0.282060] vmx: pin-based VM-execution controls
[    0.290988]      1 2 4
[    0.300072] vmx: primary processor-based VM-execution controls
[    0.309070]      1 4 5 6 8 13 14 26 28 31
[    0.318401] vmx: secondary processor-based VM-execution controls
[    0.327362]      1 3 5 7
[    0.336453] vmx: VM-exit controls
[    0.345366]      0 1 2 3 4 5 6 7 8 9 10 11 13 14 16 17 18 19 20 21 23
[    0.355252] vmx: VM-entry controls
[    0.364175]      0 1 2 3 4 5 6 7 8 12 14 15 16
[  FIRMWARE  ] Starting firmware...
[  FIRMWARE  ] multiboot header found
[    0.000000] tsc: fast TSC calibration using PIT
[    0.427451] tsc: 4200.222 MHz
[    0.439331] booting...
[    0.451246] hey 481
[    0.462772] bye 451

If you are running this in a terminal, the output consists of three parts, in three different colors.

To run xv6 as the guest OS, repeat the above two commands, but this time with KERNEL=tests/xv6/kernelmemfs. Make sure you see the following output:

xv6...
cpu0: starting
...
init: starting sh
$

Currently your lvisor is unable to run Linux as a guest OS yet. Check the project ideas if you’re interested in extending lvisor in this direction.

Virtualizing time

Recall that an OS kernel responds to user space requests through trap handlers: Upon system calls, exceptions, or interrupts, the CPU traps into the kernel from user space, the kernel then dispatches the request to specific handlers (e.g., invoking a system call or the page fault handler), and returns back to user space.

A hypervisor works in a similar way. The CPU normally executes the guest OS, such as lv6. Under certain conditions, the CPU traps into the hypervisor (in our case, lvisor), which we call a VM-exit. The hypervisor dispatches the request to a VM-exit handler, which can emulate instructions, inspect or modify the state of the guest OS, etc. Once the VM-exit handler is finished, the hypervisor returns to the guest OS, which we call a VM-entry.

We will explore this workflow by adding a VM handler to lvisor. As a starting point, since lv6 uses rdtsc to calculate timestamps, let’s extend lvisor to “virtualize” time in lv6 by intercepting rdtsc and modifying the result.

First, we need to configure lvisor to enable RDTSC exiting. In other words, whenever the guest OS executes the rdtsc instruction, we want the CPU to perform a VM-exit to trap into the hypervisor.

Now set up RDTSC exiting in lvisor: find setup_vmcs_config() in vmm/vmx.c, and add the CPU_BASED_RDTSC_EXITING flag to min:

         min = 0
                 | CPU_BASED_USE_MSR_BITMAPS
+                | CPU_BASED_RDTSC_EXITING
                 | CPU_BASED_ACTIVATE_SECONDARY_CONTROLS
                 ;

With this one-line change, run make qemu again. This time the CPU will perform a VM-exit once it hits the rdtsc instruction. However, since we don’t have a VM-exit handler for RDTSC exiting yet in lvisor, it will stop execution and dump the following information:

...
[    0.395804] *** Guest State ***
[    0.404540] CR0: actual=0x00000000e0010031, shadow=0x0000000000010020, gh_mask=0000000000010020
[    0.422030] CR4: actual=0x0000000000002030, shadow=0x0000000000002000, gh_mask=0000000000002000
[    0.439527] CR3 = 0x0000000000101000
[    0.448333] RSP = 0x0000000000120f28  RIP = 0x000000000010ba88
...
[    0.871676] vmx: unexpected exit reason 16
...

To handle RDTSC exiting, find vmx_exit_handlers in vmm/vmx.c, and add a VM-exit handler as follows:

+static void handle_rdtsc(struct kvm_vcpu *vcpu)
+{
+        uint64_t val = rdtsc();
+
+        kvm_write_edx_eax(vcpu, val);
+        return kvm_skip_emulated_instruction(vcpu);
+}
+
 static void (*const vmx_exit_handlers[])(struct kvm_vcpu *) = {
         [EXIT_REASON_CR_ACCESS]         = handle_cr,
         [EXIT_REASON_CPUID]             = kvm_emulate_cpuid,
         [EXIT_REASON_MSR_READ]          = handle_rdmsr,
         [EXIT_REASON_MSR_WRITE]         = handle_wrmsr,
+        [EXIT_REASON_RDTSC]             = handle_rdtsc,
 };

This change has two parts: a function called handle_rdtsc() and an entry in vmx_exit_handlers that dispatches to the function.

Now you have a complete VM-exit handler! In summary, the workflow is the following:

• introduce VM-exits by modifying the VMM’s configuration;
• inspect the exit reason and dispatch to a VM-exit handler;
• handle the VM-exit and return to the guest.

What to submit

This completes the exercise. In answers.txt, write up your answers to the questions (no need to submit the code). Upload the file through Canvas.

If you have done the challenge problem, include a diff in answers.txt highlighting your changes to lvisor.