LIFERAFT runs AI training on a Raspberry Pi 5 and gives a Windows standby a verified checkpoint to continue from if the primary needs to stop. A Mac runs Mission Control, where the operator can follow the job from the first alert through recovery.

Inspiration

An infrastructure alert comes in to an engineer saying a machine needs looking at. For a long-running training job, the immediate questions are “how much work can we save?” and “can a backup get it through?” We built LIFERAFT to answer both of those. It’s for infrastructure teams who want to protect their expensive compute work and give the next person on the job a very clear record of what’s gone on.

What it does

LIFERAFT resumes AI training on a standby machine from a validated checkpoint. In our current setup, the Raspberry Pi 5 runs the training job, a Windows laptop stands by to take over, and a Mac runs the dashboard. A physical ESP32-S3 button lets us inject a cooling event without actually heating the equipment. The dashboard shows which machine owns the job, where progress was saved, and whether the standby has completed a new training step.

We first proved the handoff in a campus drill with a Mac as the primary worker. The Mac saved step 25 and released ownership. A separate Windows laptop validated the checkpoint, ran step 26, and finished at step 200. That earlier Mac-to-Windows run is where we measured the 6.534-second recovery.

How we built it

  • A portable PyTorch checkpoint. We save the model weights, optimizer state, random-number state, and position in the training data. The standby verifies the file’s SHA-256 hash before loading it and records its first completed optimizer step as proof of recovery.
  • A coordinator that prevents conflicting writes. The Pi and Windows workers each connect over outbound HTTPS to a Python coordinator on Render. MongoDB Atlas holds job state and private checkpoints. A time-limited lease establishes the active worker, and an ownership version lets the coordinator reject a previous owner’s writes after the standby takes over.
  • A repeatable hardware drill. The ESP32 sends the button event over USB serial through the Windows gateway. The system waits for a verified checkpoint and a ready standby before arming the drill. Mission Control keeps the injected event distinct from recorded training progress and shows the evidence in a hash-chained event log.
  • An investigation and an operator channel. NVIDIA Nemotron calls read-only tools to inspect incident and recovery records and cite specific events. The coordinator controls recovery. Photon sends iMessage alerts and lets an operator reply with STATUS or ACK. Signed callbacks record delivery and link the acknowledgment to the incident.

Hardware setup and device roles

How we tested it

Test Observed result
Physical campus drill · Mac → Windows 6.534 seconds from incident detection to the first resumed step, 0 training steps lost at the handoff, and 200 total steps completed.
Checkpoint continuity · same-machine worker processes All four drills finished 100 optimizer updates. Each run’s recorded losses and final model tensors matched an uninterrupted baseline exactly.
Failure handling · same-machine worker processes We exercised an injected cooling warning, a primary process crash, a corrupted checkpoint, and a former primary attempting to write after takeover. The standby recovered from an earlier valid checkpoint in the corruption test.
Nemotron and Photon · hosted provider tests We matched all 8 event IDs cited in the Nemotron report against the campus incident journal. Separate Photon tests verified automatic drill and recovery alerts, signed delivery receipts, a live STATUS reply, and an incident-linked ACK.

We used an uninterrupted run with the same seed as the baseline. Comparing loss values and final weights gave us a way to check that recovery preserved the training trajectory in the controlled tests. For the campus drill, we also checked all 79 event hashes in the saved journal and traced the checkpoint, ownership transfer, and resumed step.

Recorded experiments and test results

Challenges we faced

The hardest part was handling a primary that might reconnect after the standby took over. A timed lease can let another machine claim the job, but the coordinator also needs to recognize and reject an old owner’s writes. We tested both the normal handoff and that reconnect case. Campus networking was a second constraint, so we routed coordination and checkpoint transfer through the hosted service instead of relying on the devices reaching one another directly.

What we learned

A reliable handoff needs three separate pieces of evidence. The checkpoint is intact, the standby owns the job, and the standby has completed new work. Showing those separately makes it easier for an operator to tell whether a handoff succeeded or where it stalled.

What’s next

With the Pi integrated as the primary worker, our next benchmark is repeated Pi-to-Windows drills with independently verified handoff records. We also want to measure how checkpoint frequency affects recovery time and lost work on larger workloads, using measured temperatures alongside controlled fault tests.

Built With

Share this project:

Updates

Submission history