A hypervisor is a critical component in both security and cloud computing. There is also an increasing interest in applying virtualization technologies in the area of security engineering and analysis. How can hypervisors be used to secure the existing systems? How to write custom hypervisors to perform reverse engineering and fuzzing more efficiently?

In this class, the students will learn the foundations to answer those questions, that is, the basic skills and knowledge to develop hypervisors. The class is designed in a way such that everything is built from scratch and optimized for learning. This allows students to better understand the building blocks and expand the knowledge acquired in the course by themselves afterward. Topics include UEFI architecture and programming, VT-x/EPT configuration, debugging tools, comparison of hypervisor designs, application of the technologies, and more.

This is a hands-on-oriented class. We believe that the students can learn and retain concepts and skills the best by working with those concepts by themselves and not by being taught; hence gaining the greatest value. With this philosophy, the class is designed for lab activities as the primary learning opportunities, and lectures are to explain backgrounds and the motivations for those. We will spend about half of the time for hands-on.

At the beginning of the class, students will receive skeleton implementations of our hypervisors and incrementally update them through the exercises with a clear understanding of motivations and design choices.

At the end of the class, students will also receive a full version of our hypervisor. This includes implementation of advanced concepts, such as stealth-hooking hypercall, use of VT-d, guest hardening (like HyperGuard if you are familiar with it), host hardening through CET, SMAP, UMIP, etc, handling of uncommon events like microcode update, NMI, and so on. This version can be used to complement your understanding of advanced topics and as a reference to explore more topics as you wish.

Additionally, a proof-of-concept implementation of taking and reverting to snapshots for fast full-system fuzzing will be shared as well.

Learning objectives

At the end of the class, students will gain enough knowledge and skills to start developing their pass-through UEFI-based hypervisors. Knowledge-wise, this includes but not limited to the understanding of:

• Key concepts of VT-x and programming interfaces
• General UEFI-module development workflow and tools
• Execution environment changes during OS boot and their influence on the hypervisor design
• Subtle yet critical technologies such as memory types and caches
• Application Processors (APs) start up, and emulation by a hypervisor
• “Owning” control registers and challenges with it
• Various application of hypervisors (for adding security, reverse engineering and fuzzing, for example)

Outlines

• UEFI and Hypervisors

  • Lectures: why UEFI for hypervisors, differences from OS kernel drivers, EDK2, introduction to high-level design options
  • Lab: setting up the lab environment
  • Lab: debugging firmware modules and logging output

• Basics of VT-x

  • Lectures: processor modes, VMCS, “host” vs “guest”, VM-exit, VM-entry, and snapshot and “guest” for fuzzing hypervisor
  • Lab: configuring host and guest
  • Lab: starting host and guest, monitoring CPUID execution

• OS Boot

  • Lectures: system boot phases, boot time vs runtime, physical vs virtual mode, and runtime drivers for monitoring OS activities
  • Lab: controlling VM-exits with MSR bitmaps
  • Lab: booting Windows by building host page tables, IDT and GDT
  • Lab: tracing guest page faults with exception interception and event injection

• Extended Page Tables (EPT)

  • Lectures: x64 address translation, EPT setup and activation, EPT induced VM-exits, memory types, caches, VPID, EPT-based hooking, VT-d (DMA remapping), and memory management for fuzzing hypervisor
  • Lab: reviewing address translation mechanisms with and without EPT
  • Lab: enabling EPT and tracing guest execution
  • Lab: emulating memory types

• Multi-processors Support

  • Lectures: multi-processor protocol, processor activity state, application processors startup, unrestricted guest and Hypervisor Top Level Functional Specification (TLFS)
  • Lab: virtualizing all processors
  • Lab: booting multi-processor Windows by emulating INIT-SIPI-SIPI

• Control Register Shadowing

  • Lectures: control register guest/host mask, read shadow, effects, and emulation of control register changes
  • Lab: “owning” control registers
  • Lab: booting Ubuntu by properly emulating MOV-to-CRx

• References

  • Hardware debuggers (DCI), single board computers, and relevant open source projects

Who should attend?

This class is geared towards software developers, security researchers and others with an interest in expanding knowledge of Intel VT-x, the x86_64 system architecture, and/or UEFI.

Class requirements

Prerequisites:

• Strong experience in C (or C++) programming.
• Familiarity with the x86_64 architecture such as paging and GDT
• System programming experience, such as kernel-module development, is a plus but not a requirement.;

Recommended pre-class learning materials will be introduced about prior to the class.

Hardware and Software Requirements:

The students are expected to have the following hardware and software:

• The host machine with the Intel processor, SSD, and 8GB+ RAM
• For Windows users
  • Windows 10 x64 build 19044 (a.k.a. 21H2) without Virtualization-base Security (VBS) enabled
  • Ubuntu 20.04 on WSL version 1
  • VMware Workstation Pro or VMware Workstation Player 16
• For macOS Users
  • macOS 11+
  • VMWare Fusion Pro 12 or VMWare Fusion Player 12
  • Docker Desktop
  • Homebrew
• For Linux users
  • Ubuntu 20.04
  • VMware Workstation Pro 16
• As VMs, Windows 10 x64 build 19044 and Ubuntu 20.04

The newer versions of the operating systems and other software are supported. Another Linux distro may be workable but unsupported. The other hypervisors such as KVM, Hyper-V, or VirtualBox cannot be used.

The host system can also be a cloud-provided machine if the host machine cannot be arranged locally. We recommend the student to look into:

• Scaleway’s bare metal server GP-BM1-S for a Linux host, and
• Amazon’s bare metal server m5zn.metal for a Windows host

The students are expected to complete the setup instructions that will be sent by the instructor before the class.