Inspiration
This project started with a paper about NAND flash erase operations (REO, Electronics 2025). I wanted to make SSDs last longer. While researching what actually damages SSDs in machine learning work, I kept finding the same thing: checkpoints. And then I found something better than my original idea: years of GitHub issues where people say: "My training resumes from a checkpoint, everything loads fine, and the loss is different". Nobody had built a tool that actually tested this.
What it does
FlashPilot is a tool that can be used to test checkpoints. It stops your training process on purpose, it really stops, not just pauses and starts again. It also compares the continued run to an uninterrupted control with zero tolerance (atol=0.0), using 24 deterministic checks. If the gate fails, GPT-5.6 can figure out what went wrong by looking at evidence that has been made safe, and suggest a repair that is on a list of things it is allowed to do. Then FlashPilot stops the process again and checks it again. It only reports storage savings after this (74% fewer recurring bytes in the demo).
One thing I was very careful about: GPT-5.6 never sees which failure was injected. A unit test forbids it. The model only gets the failed checks, the manifest, and the trajectory evidence and has to work out the diagnosis on its own. GPT recommends and deterministic code decides.
My favorite demo: two Hugging Face checkpoints from the same training script. Both load without an error. One resumes the same run. The other silently doesn't. You can see this pair side by side in the browser sandbox without installing anything.
How we built it
Everything was built in one Codex thread over six days, with each step being completed as it was reached: deterministic CPU workload, atomic checkpoint protocol, crash orchestration, the Recovery Gate, GPT-5.6 schemas and guardrails, HF/Lightning/FSDP/DeepSpeed qualification paths, tests, packaging. I looked at every difference, ran every command myself, and kept a build log with real outputs (docs folder in github)
Challenges we ran into
The tricky part was getting the exact recovery across different processes: the optimizer state, three RNG streams, and dropout all had to line up perfectly, or the trajectory would drift. Windows made it harder: Python can't sync a directory there, so durability is honestly labelled best effort
Accomplishments that we're proud of
The final recovery gate passes 24/24 checks with zero numerical tolerance, exact equality after a real kill and a new process. The same failure class reproduces across Hugging Face and Lightning, so the gate generalizes instead of being tuned to one trick. And the whole proof runs in the browser sandbox, where you can recompute the SHA-256 hashes yourself and corrupt a file to watch a check fail
What we learned
"Loadable" and "resumable" are different properties, and almost the entire ecosystem only tests the first one. In addition, an LLM is really helpful for infrastructure tooling, but only if you make it blind, typed and overrulable
What's next for flashpilot
Certification for spot instances, a public adapter contract so you can qualify your own training loop, and verification for checkpoint conversions (UCP/DCP) — converters exist everywhere, verifiers nowhere.
Log in or sign up for Devpost to join the conversation.