ORACLE LENS

Research that explains itself honestly — at your level, not the author's.

Comment

Inspiration

A few months ago I made it to the semifinals of a global-scale quantum mechanics research challenge. Round 2 handed us a real research paper and a set of questions we had to answer from it — not summarize, actually reason from. I threw the paper at every AI tool I had. Every single one gave me an answer that sounded right and quietly assumed I already understood half of what the paper was doing. Not wrong, exactly — just calibrated for someone who didn't need the help I needed. I ended up qualifying, but it took hours of manually cross-referencing terms and rebuilding context the hard way. If something like ORACLE had existed then, that round would have taken a fraction of the time.

That's the whole reason this exists: an honest on-ramp for real research, at whatever level you're actually starting from.

What it does

ORACLE answers satellite and space-science questions three different ways depending on what the question actually needs — a real-time orbital lookup (SGP4 propagation off live CelesTrak data), a literature synthesis (arXiv + Semantic Scholar, credibility-scored, not just keyword-matched), or, when nothing existing answers the question, a generated Python simulation that has to survive its own physics-verification and red-team falsification pass before it's allowed to count as an answer.

LENS then takes that result and re-derives it at five comprehension levels — from "explain it like I'm 6" to full researcher register — with a deterministic verifier checking that every number in the simple version still traces back to the same evidence as the expert one, and that the confidence label never gets rounded up just because the explanation got simpler.

How I built it

It started as something closer to a researcher-teacher hybrid than a pipeline — I got computational mode and literature mode working in a couple of hours each, mostly because SGP4 propagation and literature retrieval-and-synthesis are well-trodden problems with a lot of existing scaffolding to lean on.

Empirical mode is where it stopped being easy. Getting ORACLE to generate a correct simulation, run it safely, and know whether to trust the result turned into the majority of the build time.

Challenges I ran into

The honest version of "challenges" here is a long list of real bugs, most of which are now locked in as regression tests so they can't silently come back:

  • Sandbox failures that weren't code failures. Some of the earliest generated experiments would fail to run at all in the isolated subprocess — not because the physics was wrong, but because the sandbox itself needed tighter resource and execution constraints before I could trust its output either way.

  • A conjunction-assessment bug that was wrong by a full order of magnitude. methods/conjunction.py computed a single shared propagation epoch for both satellites instead of propagating each one to its own — a subtle enough mistake that the code ran clean and produced a plausible-looking number that was just wrong.

  • The exact same class of bug showing up twice, independently. While fixing that epoch issue in conjunction I found the same km/m threshold inversion sitting in both agents/physics_verifier.py and core/physics_validators.py — two independent gates that were supposed to catch bad physics, both quietly wrong in the same way.

  • A regex that parsed the wrong number entirely. The experiment generator's mass-parsing pattern in agents/experimenter.py matched any number followed by "kg" — so a question phrased around a mass mentioned earlier in the sentence, not the one meant for the calculation, silently fed the wrong mass into the simulation.

  • A confidence system that lied about being honest. Early on, claim_ledger.py marked every sentence in an answer as "supported," regardless of whether there was actually evidence behind it — which meant the system could look rigorous while not actually checking anything.

  • Scientific notation that lost its sign. OracleConductor's number extraction parsed 1.5x10^-4 by splitting on the literal string "10" and stripping the minus sign along with it — turning a very small number into a very large one, silently.

  • A validator that only checked for the errors it already knew about. FalsificationEngine.check_numerical_artifacts accepted an expected range as a parameter and then never actually used it — so a finite-but-astronomically-wrong result (exactly the kind the conjunction bug above was producing) sailed straight through as "not NaN, not infinite, therefore fine."

  • And once, everything ran perfectly and was still wrong. No crash, no malformed output, no validator complaint — the simulation executed cleanly and returned a physically implausible answer that just hadn't been checked against the right constant yet. That one didn't get fixed by better code. It got fixed by adding another layer that didn't exist before.

Every one of these got turned into a named regression test the moment it was found and fixed, specifically so the bug-hunt itself became part of the system's honesty guarantee — LENS can only be as truthful as the evidence graph underneath it, and that evidence graph only got trustworthy by repeatedly catching itself being wrong.

What I learned

The empirical layer taught me that "the code ran without errors" and "the result is true" are almost unrelated claims, and that most of the interesting failure modes in an AI research system don't look like bugs — they look like confident, plausible-sounding answers. Every fix in that list above added a layer that didn't exist before: a second physics gate, a numerical-artifact check with teeth, a claim ledger that actually tracks support instead of assuming it. The system got more honest by getting caught being wrong, repeatedly, and I don't think there was a shortcut to that.

What's next for ORACLE + LENS

LENS's own modules — the five-level derivation, the verifier, the grounded chat — are currently validated through live end-to-end runs rather than dedicated pytest coverage the way the core pipeline is (249 tests and counting). Bringing LENS up to the same testing standard as the research pipeline underneath it is the next concrete step, so the honesty layer is held to the same bar as the research it's explaining.

Built With

Share this project:

Updates

Submission history