Project: Fuzzing For Garbage Collection

Overview

Fuzzing is a surprisingly successful technique for identifying security vulnerabilities in code. Simplistically put, it takes a program and then repeatedly executes it, applying randomized inputs to it to try to get all parts of the program to execute, in the hope of discovering a code path that leads to the program crashing, and thus a potential security exploit. Although the approach sounds implausibly inefficient, with intelligent selection of inputs and many other sophisticated techniques, fuzzing can be extremely effective and has become an very important tool for identifying security vulnerabilities in programs, and is widely used by major software vendors such as Microsoft and Google.

Although important runtimes such as V8 are aggressively fuzzed by their vendors, they are fuzzed at the application level and there are not systems that specifically fuzz the GC code. This project will explore directly fuzzing the garbage collector. Since this has never been done before, it is not obvious how best to do this. One simple approach would be to use a DSL to write code that directly accesses the GC’s api, and the fuzz that code in order to explore the GC effectively. It turns out that many years ago, Robin Garner wrote such a DSL for Jikes RVM / MMTk, which he called the “GC harness”. His goal was not fuzzing (the technique was not well known then and security was not a first order concern then), but rather for writing tests that would exercise different parts of the GC framework independently of any particular runtime.

Themes

Garbage collection
Security

Requirements

Coding MMTk is written in Rust, so the fuzzing framework should be written in Rust or a suitable scripting language or DSL.
Fuzzing You are not expected to know anything about fuzzing before starting the project, but I provide some references below so you can make yourself familiar with the key ideas.

Project Length

This project is very scalable, and can easily be adapted to suite a student’s needs. A one semester project is quite possible. The project has sufficient depth that it could form the basis for an entire PhD.

Project Outputs

This project should produce a fuzzing framework (however modest) that helps improve the security of MMTk. The work will be very interesting to many parties, so a productive one-semester project may well lead to a publication.