Inspiration
Reinforcement learning is now the main way AI models get better, and its bottleneck is the grader, the code that decides whether a model got a task right. Today, graders run right next to the model being trained, and models learn to game them: METR caught OpenAI's o3 reward-hacking in 30% of runs on one benchmark. Training environments are also shipped as files, so they leak, get resold, and their creators get paid once.
We asked: what if the grader were sealed, so a model can't reach it, a creator can't leak it, and nobody, including us, can fake its scores? The Solana Foundation's essay on AI and the launch of Solana Confidential Compute pointed to the missing piece: graders whose work is verifiable onchain, and creators paid every time their work is used.
What it does
Modélmon is a marketplace of sealed RL training Gyms on Solana.
- Sealed grading. Each Gym's answers, hidden tests and grader live in a sealed grader. Trainers keep their own model and GPUs, send attempts to a grading endpoint, and get back scores, never answers. The held-out test set is chosen at random inside the grader, so nobody picks it.
- Pay per rep, at RL speed. Every grading call is gated by x402. A trainer opens a Solana payment channel with one USDC deposit, which also acts as a hard budget cap, then pays each batch of attempts with a signed running total. One settlement transaction pays the Gym's creator, its contributors and the platform, and refunds the rest.
- Only verified graders count. Our Solana program accepts a payout, a score or a new Gym only if it is signed by a grader with a live onchain record for approved code. Changes to that approved code are public and timelocked. When we deliberately swapped in a tampered grader that inflates every score, Solana refused it three times onchain.
- Proof of improvement. After training, the sealed grader tests the model on held-out tasks it never saw, and writes the before/after result onchain.
- Live monitor. A three-lane view (Trainer → Sealed grader → Solana) shows real training runs as they happen, with links to Explorer.
Real results (devnet, every reward paid over x402):
- Qwen2.5-1.5B, trained with GRPO on our Transaction Guard Gym (spotting unsafe Solana transactions an AI agent is about to sign): 4% → 60% on held-out tasks. The rate at which it would sign unsafe transactions dropped from 97.8% to 47.8%.
- Cloudflare's Clef Flash, where we trained only its decision head, on our Reward-Hack Judge Gym (catching coding agents that try to game their evaluations): 58% → 85% on held-out tasks. On the real, public SafeAgents-Judge-Bench, its rate of wrongly blocking legitimate agent actions fell from 49.6% to 38.4%.
How we built it
- Solana program (Anchor, deployed on devnet):
- Gym registry and grader records
- an approved-code list with a timelock
- payment channels whose vouchers are checked by Solana's built-in Ed25519 signature verification
- an automatic royalty split at settlement
- onchain improvement records
- x402 grading API (Next.js): HTTP 402 challenges, with payment-channel vouchers in the X-PAYMENT header.
- Sealed grader: Gym bundles are encrypted on the creator's machine to the grader's key, which the creator reads from its onchain record rather than from our server. Sealed storage is encrypted at rest. Three deterministic Gyms: Solana transactions executed in LiteSVM, the Transaction Guard Gym, and the Reward-Hack Judge Gym.
- Training:
- Qwen: GRPO with TRL and LoRA on Modal GPUs.
- Clef Flash: REINFORCE on its decision head, with a frozen and cached 9B backbone.
- Every reward in both is paid over x402 through a public tunnel to the grader.
- Attestation: Modélmon is designed around Solana Confidential Compute. We applied for access through SCC's builders program but didn't receive it in time for this project, so an attestation authority writes the grader records for now. Those records carry the same fields SCC records (key, code fingerprint, hardware type, expiry, a fresh slot), and our Solana program still enforces approved code, report freshness and expiry. Switching to SCC's verifier is mostly a change of which program writes the record.
Challenges we ran into
- Getting GRPO to learn. Small Qwen3 models gave the same answer 16 times out of 16, which leaves GRPO nothing to learn from. We probed six models for answer diversity and switched to Qwen2.5-1.5B.
- Training bugs: right-side padding and gradient checkpointing turned answers into gibberish, and one run collapsed onto fixed answers until we added a KL penalty.
- A Gym that was too easy. Clef scored 85% on our first Reward-Hack Gym, so we redesigned it around Clef's published weaknesses.
- Catching our own mistake. A sealed test once scored the wrong model, because a stale cloud container was still answering. We now verify the model's weights hash on every answer, and we documented the mislabeled onchain record instead of hiding it.
- Performance and the first real settlement: a stack overflow in the program, and a quadratic base58 encoder that froze the server.
Accomplishments that we're proud of
- Real, verified improvements recorded onchain: 4% → 60% for Qwen and 58% → 85% for Clef Flash, plus measured transfer on a real public benchmark.
- The tamper demo: a modified grader was refused by Solana itself (CodeNotApproved, then GraderNotVerified twice), and the real grader settled the same channel a moment later.
- The full loop on devnet: a trainer opens a channel, thousands of paid reps follow, and one transaction settles everything with the royalty split.
- We did this project in 24 hours!
What we learned
- Whoever controls the grader controls the truth. Isolating the grader from the model is only half of it; the other half is making the grader's operator, including us, unable to fake results.
- RL needs variation. A grader is only useful if the model sometimes gets tasks right and sometimes wrong. Picking the right task difficulty and model mattered as much as the algorithm.
- Honest measurement is a feature. Weights hashes, sealed held-out splits, and failed runs recorded alongside the successful ones are what make "proven improvement" mean something.
What's next for Modélmon
- Run the grader inside a real TEE (Intel TDX on Phala), and switch grader records to Solana Confidential Compute's onchain verifier once its program is public.
- Let any creator seal their own Gym by supporting the standard verifiers environment format, with royalties on every rep.
- Escrow-backed exclusive windows for labs, and expert-domain Gyms (finance, legal, security).
- Make improvement records the standard proof an AI agent shows before it's trusted with money.
Built With
- anchor
- next.js
- phala
- solana
- x402