SmartOptic

Inspiration

We had already built VisionCheck, a phone app that screens children for vision problems. It works the way almost every acuity test works: show letters, get smaller, stop after two mistakes, and report the line above. A staircase rule.

Then we watched what actually happens when a child takes it. Some of them stop trying. They tap randomly, or they guess, or they get bored four letters in. And the staircase rule has no idea. It takes "this child answered wrong" and returns a number with total confidence, the same confidence it would have for a child who genuinely couldn't see.

That's not a small bug. In screening, a wrong number means a referral that wastes a family's day, or a child who needed help and didn't get flagged.

So we wanted to test a different method. And that's where the second problem showed up.

We tried using AI research assistants to survey the literature, and we kept catching them inventing citations. Plausible authors, plausible journals, PMIDs that went nowhere. If a tool will fabricate a reference, we can't trust anything else it tells us either. A research tool that quietly invents things is worse than no tool, because it's confidently wrong at scale.

SmartOptic came out of those two frustrations at once. A system that runs a real research cycle on screening data, and where the honesty is enforced in code, not promised in a system prompt.

What it does

SmartOptic runs a full research cycle end to end, in about forty seconds, and writes a report you can check.

Literature

SmartOptic searches PubMed, retrieves real papers, and pulls open-access full text via Firecrawl. Every claim in the output has to be tied to a PMID that was actually retrieved.

If the model produces a citation that wasn't in the retrieval set, that citation is dropped, and any claim left unsupported is dropped with it. Both are recorded in the report.

Hypothesis

The system proposes something testable. If the Analyst can't actually execute it, it's rejected before it ever touches the data. No hypothesis that can't be run.

Analysis

SmartOptic scores 800 simulated eyes twice, from the identical recorded answers. Once with VisionCheck's staircase rule, and once with a Weibull psychometric function fitted by maximum likelihood over the whole response stream. It then bootstraps the difference for a confidence interval.

The important part is what happens when the fit is uncertain. When the fit's confidence interval is wider than 0.6 logMAR, it reports "unreliable" instead of converting a bad fit into a number.

A deferral is never counted as correct. The psychometric method gets credit only when it commits to an answer and that answer is right.

Verdict

This is the piece we care about most.

The model is never asked whether the hypothesis was supported. A pure function reads the confidence interval, the sample size, and the deferral rate, and returns the status.

The model is told the status and asked only to write prose explaining it. If the prose contradicts the status, the prose is discarded and the discard is recorded.

Dossier

The system produces a markdown report with the chart, the numbers, the references, and a limitations section the system writes about itself.

On our run it returned SUPPORTED. The psychometric fit removed 29.8 percentage points of confident misclassification, with a 95% confidence interval of [+26.5, +33.0].

But the report doesn't stop there, and this is the part we'd point a judge at. The fit declined to score 41% of sessions.

So the dossier also reports the like-for-like number. Restricted to the eyes both methods scored, it's 9.9% versus 4.2%. A much smaller claim, printed at the same size as the flattering one.

And on disengaged children, where the fit deferred 98% of the time, the system flags itself: "the fit's low error rate here means it mostly refused to answer, NOT that it measured these children accurately."

We didn't write that caveat into the report. The Analyst generates it when the deferral rate crosses a threshold.

How we built it

We built SmartOptic in Python, with a hard architectural rule: the model is a writer, never a judge.

Every number in the final report is computed by code. The LLM never sees a place where it could originate a fact. It writes prose about facts that already exist.

The main pieces are:

Pydantic schemas that reject impossible data at the boundary, such as a negative trial count or a confidence interval that doesn't contain its own point estimate.
A psychometric fitting module using Weibull, maximum likelihood, a guess rate fixed at γ = 0.25 for four-alternative choice, and profile-likelihood confidence intervals.
Bootstrap resampling for the difference between methods.
A JSONL trace, flushed per line, that records every stage, including the failures.
A Streamlit console that tails that trace live, so you can watch the stages complete or degrade in real time.
Around 290 tests, several of which exist specifically to fail if someone weakens an honesty guarantee.

That last category matters more than the count.

There's a test that fails the moment anyone enables the "use my real screening results" control without wiring the actual data path. A control that looks like it uses your data but silently runs on simulated data is exactly the failure this project exists to prevent.

Every method that can't produce a trustworthy answer returns None with a stated reason, rather than a number that looks fine.

Challenges we ran into

Deciding what "better" means

This was the real one.

The psychometric fit looks spectacular if you count its refusals as wins: 32.2% error down to 2.5%. But it only answered 59% of the time. The staircase rule answered every time.

Comparing those two numbers directly is a lie, and it's the easy lie because it's the flattering one.

We ended up making the code report both, always, at equal prominence. We also added a guard: if the deferral rate hits 50%, the verdict is forced to inconclusive no matter how good the effect looks.

You don't get to win by abstaining.

The model provider failed mid-demo

Our Featherless model got gated, a live 403 in the middle of a run.

The literature stage had already retrieved 14 real references, so the system had a choice: fabricate a summary, or ship the references with no synthesis and say why.

It wrote the exact error into the report's provenance notes and returned the references unsummarised.

We decided to demo it that way rather than swap the model. A system that degrades loudly is the entire pitch.

PubMed ANDs every search term

A well-phrased research question can return zero results. Nine content words can get you nothing.

We built a broadening ladder that drops trailing terms until something matches, and then reports which form actually ran, because a silently rewritten query is a silently different question.

We also learned the hard way that "psychometric" is ambiguous in PubMed. It matches questionnaire validation studies as readily as psychometric functions. Half our early results were about survey instruments.

Template bugs that only appeared with real data

Jinja's whitespace control silently merged references into one another, but only when there was more than one citation.

Every unit test passed. We only caught it by reading actual rendered output.

Four separate whitespace bugs were invisible to the tests we'd written.

A silent test-collection failure

Our schemas define a class called TestRecord, so pytest's class collection is disabled in this project.

We wrote 45 tests as classes and pytest cheerfully collected zero of them and reported success.

Accomplishments that we're proud of

The verdict is a function, not a sentence

decide_status() takes the hypothesis and the analysis and returns the status. We can point at the line.

No prompt engineering. No "please be accurate." No trusting a model to be honest under pressure.

If the confidence interval includes zero, the answer is inconclusive, and nothing in the system can override that.

The system argues against its own result

The most compelling line in the dossier is the one warning you not to over-read the headline.

It's generated by the same code that produced the headline.

It reported its own outage in the deliverable

Not in a log file.

In section 5 of the report, with the error text, where a reader will see it.

Simulated data is labelled everywhere it appears

The system labels simulated data everywhere it appears, including a banner explaining that the simulator generates responses from a psychometric function, which is the model under test.

So the result measures whether the estimator recovers a known ground truth, not whether it works clinically.

That's a real limitation, and it's stated at the top, not buried at the bottom.

What we learned

Honesty is an architecture problem, not a prompting problem

Every time we asked a model to be careful, we got carefulness-shaped text.

Every time we made a wrong answer structurally impossible, using a schema that rejects it, a function that owns the decision, or a filter that runs unconditionally, we got an actually correct answer.

The fix is always upstream of the prompt.

The flattering number is usually the wrong number

+29.8 percentage points is real, and it's also not what a clinician needs to know.

Building the paired comparison into the report meant building something that makes our own project look weaker.

That turned out to be the feature we'd defend hardest.

"I don't know" is a valid scientific output, and most systems can't say it

Getting the psychometric fit to return "unreliable" instead of a plausible number was more work than getting it to fit curves.

Refusal has to be designed in.

Tests can pass while the thing is broken

The template bugs, the zero-collected tests, and the headline that was technically true all passed automated checks.

We found every one by reading real output.

What's next for SmartOptic

Closing the loop with VisionCheck

The parser for real screening exports already exists and works.

What's missing is the wiring. Right now the pipeline generates its own population.

That's why the "use my eye screening results" control in the UI is visibly present and visibly disabled, with a test that fails if anyone enables it before the data path is real.

It'll ship as an honest gap rather than a fake feature.

A triage agent

Turning a verdict into a recommendation a clinic could act on, and just as importantly, refusing to when the evidence doesn't support one.

Validation against real ground truth

Everything here runs on simulated eyes, because no VisionCheck session currently carries a clinically measured threshold to score against.

Until that exists, this measures internal consistency, not clinical validity.

We'd rather say that plainly than imply otherwise.

Generalising the pattern

Nothing about the honesty architecture is specific to vision.

Code-owned verdicts, verified citations, deferrals that never count as wins, and degradation that announces itself would work for any domain where a confident wrong answer costs more than no answer.

Which, in medicine, is all of them.