Trust Me Bro

our chat client
a UML-esque diagram of the network
Happy Path screenshot

Inspiration

AI is currently the largest economic sector in the world and yet there it's almost entirely based on blind trust. Users assume that by paying for a given model all their requests will be executed by said model, but the company providing the model has accountability in ensuring this and in fact sometimes may have incentives to do otherwise, be it for load balancing purposes, anti-competitive reasons or otherwise. We propose a cryptographically verifiable method of ensuring that the model we expect to be ran is in fact ran.

What it does

Trust Me Bro is a system that allows AI model providers to prove that each inference request was executed by the model and hardware they claim to be using. By combining hardware-level attestation (Trusted Platform Module "TPM" includes hardware private keys that we can get the Integrity Measurement Architecture "IMA" to use to sign audit logs about system state) with runtime profiling of GPU power draw, VRAM usage, and process integrity, we can verify that the usage statistics we receive have not been tampered with. The result is that end users, auditors, or partners can trust that “Model X” actually ran on the stated hardware — not a cheaper, spoofed, or downgraded model.

How We Built It

Our actual project is a small part of a larger architecture since we sit between an LLM provider's load balancer/router and the processes that actually run the inferencing. We ask that the (otherwise unchanged) network requests are routed through a process we run on their system.

The rest of the code we wrote was building up an example infrastructure around that, with a LLM client, load balancer, models to test with and our server. The LLM client, load balancer and model are just sample trivial implementations, mimicking how an LLM company would set up their infrastructure.

The client, which are the users of the LLM server, will send their query to the LLM server and then query our server (the "TMB" server) to verify the output. After receiving a request from client the TMB server continuously listens to an incoming data stream providing us information about the system that is currently doing the inference for our request, such as GPU utilization, VRAM usage, power draw, etc. Given the diagnostic data, they are fitted to beta and bimodal distributions using maximum likelihood estimation and Newton methods, which we can check if new data is likely to be from the distribution as a method of determining whether or not we are given adequate data. We can verify that these have not been spoofed by verifying the audit log we recieve from the datastream, signed by an on-die private key in the TPM using IMA. For maximal efficiency, this is compiled and run on a separate C file triggered by Python's subprocess library. The final verification result is then sent back to the client.

Accomplishments that we're proud of

The hardware attestation is incredibly complex and Max was working on it constantly and he has figured out how to verify the signature of massive object logs just by staring at the documentation and about three hundred thousand signatures for basically the entirety of the Hackathon.

We're also proud of how well we built our ultra-interconnected network such that we can mostly mimic the complex architectures of load balancers race-condition free between three different languages and how easy we made it to set up an integration testing environment--if it hadn't been as testable as it is we would never have finished.

We're also proud of the low barrier of entry it would take for an LLM company to adopt our system. All they have to do is send all the routing requests they would normally send to the model instead to our tracing client, as well as ensure they store a minimal amount of metadata that they store already (unique ids for each inference request). No need to modify model logic or load balancing logic.

What we learned

C is really bad for networking. We initially tried to write the tracing client in C (since it needs to be statically linked to verify the hash) but man string manipulation and networking primitives are terrible in C. Rust is also bad compared to something like python, but still significantly better.

We learned a whole lot about TPM, IMA, PCRs, TEE, Secure boot and many other three-letter acronyms that in practice just look like massive files full of hashes. It takes a lot of cryptography to ensure a pc is executing signed code from boot throughout execution.