Lux Max

Main Dashboard showing a run
The most important view, a live run view of all the attempts made
Papers It has Researched Through
The protocol we are scoring against (meant to be portable to other domains)
Internal checks
game plan of the run
categorises the patterns to weak /promising / banned
Estimate the delta, and then compare it to actual delta. Good for future evaluations
Overall Health Check of the runs
How much we spent thus far
How autonomous was our run, and what happened is traced

Inspiration

Read the NOVA and AGENTX papers from Tencent and KuaiShou respectively, along with a few others. Just had so much fun implementing theory into something more tangible, and seeing the number go up during the run is so cool.

What it does

Point it at KuaiRand-Pure and walk away. For up to six hours it invents experiments, writes its own code changes, trains and scores them, and decides which improvements are real.

Ideas that fail become lessons it reads before proposing again and ideas that "win" must survive repeat tests and all idea requires an explanation: a gain that can't show why it happened is refused.

What comes out the other end is a validation-best submission in the kit's schema, a complete per-iteration log of every attempt (the diff, the scores, the errors, the recoveries), and a live dashboard that tells the run's story honestly

Zero manual interventions on the scored run.

How we built it

All our design choices are mostly inspired by NOVA, AGENTX, with a few from AIDE and AIRA etc.

The harness is a pipeline of five decoupled stages that only talk through an append-only event log. Firstly, a hypothesis queue seeds from a cited idea bank and refills from an LLM researcher that reads the run's own lessons file
an LLM coder turns the chosen idea into a git-committed diff against the candidate model (with a diff sanitizer, one error-carrying retry, and a full-file fallback so ideas die on science)
A sandboxed runner executes each candidate under an environment whitelist that physically withholds the test labels. We also have a verification cascade (free rule checks → model read → quick test) so cost dont explode, and survivors face 3-seed repeat tests.
A measurement ladder then screens on one seed, replicates on three paired seeds against a noise-calibrated bar, and holds promotions behind two final gates: a budgeted unbiased wall (the random-exposure log, which the search cannot train on) and an attribution check, in which a win must move the observables its own hypothesis declared, or it isn't accepted.
Failures are written into a memory the proposer reads next round and forbidden patterns are never retried. 6.A live dashboard renders the whole run honestly, and every number carries measured-vs-forecast provenance.

Built on Claude Code CLI (Sonnet researcher, Haiku coder) and FastAPI

Challenges we ran into

Accomplishments that we're proud of

The proudest accomplishment is translating something theoretical into something practical. The papers did not give exact implementation steps / architecture. We have to figure out how to implement it from the theory, which was a fun exercise.
Maybe the next produest moment is the measurement ladder catches every fake win. Fake in the sense that both early screen "gains" were single-seed mirages that evaporated at 3 seeds. The most important addition was an attribution gate (a win must move its claimed observables). Caught a lot of noise.
Another accomplishment we are proud of is the full autonomy of the run, we did not intervene once at all.

What we learned

Most important thing is we found out noise is the default explanation. Every single-seed "win" we celebrated evaporated under three paired seeds. On this task, a delta under ~0.01 from one seed means nothing, and a yardstick for evaluating whether its legit or noise is super important.
The decision to have an append-only event log from the start saved us a lot of time. It started as debugging hygiene and became the product, catching our frontend lying, proving our interventions count, generating our iteration logs for judging, and made the public dashboard trustworthy.
Infrastructure is the super important. More than half of our failed attempts died on plumbing (a diff that wouldn't apply, a stripped newline).
Until we made recovery a first-class, logged behavior, our most sophisticated stage (the researcher) never even got to run.

What's next for Lux Max

Really let the researcher off the leash. The throughput fixes finally make the queue drainable, so the next milestone is runs where most hypotheses are agent-proposed, cited, and lesson-driven. Right now, we have a bank of a few ideas the agents can try, and if it all fails, then the agent searches the web.

What we think could be super exciting is prompt evolution on the proposer (GEPA-style, at a fraction of the rollouts), a wider model-class search now that trees and deep-cross architectures are in the bank, and porting the harness to a second task family to prove the protocol ( hidden walls, attribution, noise-calibrated bars) transfers beyond recommenders.