Strata - a learning diagnostic that drills backwards

Inspiration

I got 41 on a calculus test. My tutor spent three weeks re-teaching calculus. I got 47.

The problem was never calculus. Months later I watched my younger cousin write 3/4 and say "three little pieces", and realised I had been saying the same thing in my head for nine years. I never actually believed a fraction was a division. Rational functions, the quotient rule, integration by parts — all of it was resting on a sentence I had quietly gotten wrong in grade six.

Nobody teaches backwards. Every tool I have used answers the question in front of you. None of them ask when your understanding actually broke.

What it does

Take a photo of your marked homework. Strata reads the page, ignores everything you got right, and takes a question you got wrong.

Then it goes the other way from every other study tool:

Map — it builds the prerequisite graph beneath the skill you failed, typically forty to sixty concepts reaching back to grade four.
Drill — roughly six questions, twenty seconds each. You never see the graph. You see one question and a core log filling in beneath you, labelled by grade: 12, 10, 8, 7, 6.
Fracture — it names the root gap and shows the damage chain: every concept that has been quietly unstable since, ending at the question you failed this morning.
Repair — it rebuilds bottom-up. Each step is one sentence that contradicts the wrong idea you actually revealed, an interactive simulation you drag, and a check. The slider stays locked until you commit to a prediction.
Prove — it hands back the exact question you failed. Same question.

You can also just type a question — anything from arithmetic to calculus, or any programming question. It is classified, then matched against the graph by embedding, which takes two to four seconds when the skill is already there. When it is not, the graph extends upward to reach it and the extension is written back — about ten seconds, once, ever. The first student to ask about determinants pays for that; everyone after them gets it instantly. It is not five canned questions with a text box next to them.

Similarity alone could not decide those matches, and finding that out is most of what the matching logic is. Across fourteen real questions the one clear mis-match — "why is the derivative of sin x cos x" landing on the derivative of eˣ, because the graph holds no trig derivatives at all — scored 0.70, while correct matches scored 0.67, 0.73 and 0.74. The wrong answer sits inside the range of right ones, so no threshold separates them: cosine knows which node is nearest, not whether that node covers the question. Above 0.80 we take the nearest node; below 0.66 we grow the graph; in between, one cheap call asks which of the top three the question actually requires, or none of them.

And there is a classroom view: the same diagnosis across a whole cohort, on one cross-section. Twelve of twenty-four students broke in grade 6–7, most of them on the same idea, six years before the material they are failing now — which is a different fact from "some of them are struggling", and a different lesson plan. Those are real scans of a simulated cohort: the engine, the stored probes and the answer noise are all real, and every verdict is stored as it came out, including the ones placed on a neighbouring concept. Aggregating invented numbers would have made it a mockup of a feature rather than the feature.

Why backwards

Re-teaching the thing you failed works when the failure is local. It cannot work when the failure is downstream of something older — and that case is not rare, it is the one every tutor recognises and no tool addresses. The weakest-link model is that intuition written down: performance at a skill is capped by the shakiest idea underneath it, so no amount of work at the top raises the ceiling.

Two design decisions follow, and both are pedagogy rather than engineering.

The learner never sees the graph. A sixty-node prerequisite DAG is the right internal representation and the wrong thing to show someone who has just been told their foundation broke. They see one question and a depth reading, labelled by the grade they were in.

The simulation stays locked until you commit to a prediction. A manipulative you can touch immediately is a toy: you fiddle, it moves, you learn nothing. Committing first turns the same manipulative into an experiment, and being wrong on purpose is where the work happens — the predict–observe–explain pattern from science teaching, applied to a fraction bar. Nothing is scored, and the wrong prediction is not punished; it is the thing the next sentence is written against.

The belief model is standard Bayesian Knowledge Tracing (Corbett & Anderson, 1995) with the learning rate pinned to zero during diagnosis: we are measuring what is there, not crediting the learner for improving while being tested.

A diagnosis costs the learner about two and a half minutes. The alternative it replaces — a tutor re-teaching the wrong chapter — costs weeks, and that is what makes this worth building rather than interesting.

How we built it

Four layers. Only two of them call a language model.

Layer 1 — Prerequisite graph synthesis (model)

Gemini generates the prerequisite DAG through structured outputs. Generated graphs hallucinate, so three guards run on every one, and each catches a failure that corrupts the diagnosis rather than merely looking untidy:

Cycle rejection — a prerequisite relation cannot contain one, and the engine cannot topologically sort a graph that does.
Grade monotonicity — a prerequisite taught after the thing it supports inverts the direction of the search.
Embedding dedupe — "basic fractions" and "fraction concepts" as two nodes split the posterior between them, and neither ever crosses the confidence threshold.

Layer 2 — Belief network (no model)

Each node carries two quantities: P(L), whether you hold that idea, and P(C), whether you can perform it. They are related by weakest link:

P(C_v) = min( P(L_v),  min P(C_u) )     over every prerequisite u of v

One broken idea deep in the graph caps everything above it, however solid each of those skills is on its own. That cap is what learning debt is.

This started as a product, P(Lᵥ) · ∏ P(C_u), which reads better and is wrong: it decays geometrically with chain depth, so a grade-11 skill on an eight-level chain hit the clamp floor before any evidence arrived, for every learner. Three things downstream were silently reading a saturated constant.

Answers are noisy, so they update beliefs through Bayesian Knowledge Tracing (Corbett & Anderson, 1995), with the guess rate measured from the probe's own option count rather than assumed. One detail we are proud of: during diagnosis the BKT learning rate is pinned to zero. We are measuring, not teaching — a non-zero rate lets the model conclude the student is improving simply because they answered questions.

Layer 3 — Entropy-greedy probe selection (no model)

Asking all forty-seven ancestors is not a product. Under downward closure the ancestor set is a partial order, and every partial order is binary-searchable: answering correctly at v eliminates everything beneath it, answering incorrectly collapses the candidate set to v and below. It is git bisect, run on a DAG instead of a commit history, with noisy observations instead of certain ones.

Each probe maximises information gain:

v* = argmax [ H(p) - E[ H(p | observation at v) ] ]

which peaks when the probe halves the posterior mass.

Layer 4 — Probe and lesson authoring (model)

A probe must isolate exactly one node, and every distractor is a named misconception, not filler. So which wrong answer you pick does not merely say wrong, it says wrong in this particular way — and that misconception id is what the repair lesson is written against. Two students who fail the same node receive different lessons.

Probes that mention code carry the code. Repair simulations are hand-built React components; the model picks a template and fills its parameters, and is allowed to answer "none" when nothing in the catalogue honestly fits.

Challenges we ran into

Letting a student say they don't know. On a screen that deliberately hides right-or-wrong, someone who does not know is otherwise forced to guess — unpleasant, and it puts noise into the one place a diagnosis cannot afford it. Declared uncertainty is now its own observation, and a stronger one: a wrong multiple-choice answer is consistent with a quarter-chance guess that missed, whereas nobody declines to guess about an idea they hold. Read at a guess rate of 0.01 rather than 0.25, it moves P(hold) to 0.26 where a wrong pick moves it to 0.48.

Making a graph legible without showing it. The first build rendered the full 60-node DAG. It looked technical and was useless — testers closed the tab. The rule that came out of it governs the whole product: hide the tree, show the drill. The learner sees one question and a number going down.

The dark theme was half right. Obsidian suited the drilling and fought everything after it. Repair is ten minutes of reading and dragging, and the palette was telling students "this is an analysis tool" when it needed to say "sit down and work". It is now a 1960s geological survey sheet — buff stock, flat spot inks, heavy rules — which is the subject's own visual world rather than a theme applied over it.

Four modelling bugs that only measurement found. Doubt propagating down the full ancestor cone destroyed the very foundation the true root needs intact to score, so diagnoses slid to the bottom of the graph. posterior / support could exceed 1, so a wrong answer pinned a node's skill estimate at the ceiling. A normalised posterior over node hypotheses always sums to one, so a learner who answered everything correctly was still handed a confident fracture. None of these are visible by reading the code; all of them were obvious the moment we simulated learners and read the numbers.

Accomplishments that we're proud of

The diagnosis runs with the model switched off, and that is checked rather than asserted. npm run verify:offline deletes the API key from the process, forces offline mode, and drives a full scan on both subjects, failing if a single question had to be authored live. Both locate their planted gap exactly, in five questions, with zero live authoring.

Every load-bearing claim has a script or a test that fails if it stops being true, and several were false when we first went looking. /api/health reports what a deployment can actually do — database, seeded subjects, cohort, and the sample photo fetched exactly the way the app fetches it — because the difference between a laptop and a deployment is the worst place for a bug to live, and we shipped one there: the sample image was read from public/ on disk, which works locally and fails in production because those files are served by the CDN and are not on the serverless function's filesystem.

The numbers in our docs are measured, not aspirational. Over 400 randomised runs per planted gap:

subject	exact	exact or adjacent	probes
Mathematics, mean over 14 gaps	65%	85%	6.5
Computer science, mean over 10 gaps	71%	90%	5.5*

* The probe count is measured across five of those ten gaps and ranges from 5.2 to 6.0; the accuracy columns are over all ten.

That gap between the two columns is why the classroom view leads with a grade band rather than a concept: naming the exact node is right about two thirds of the time, naming the right bed is right about nine tenths, and a teacher needs the reliable claim.

Eight noisy probes cannot pin one node in forty with certainty. That is an information bound, not an implementation gap, which is why the tool reports its confidence instead of asserting a single answer.

What we learned

Prerequisite structure is a partial order, and every partial order is binary-searchable. Once we saw that, a problem that looked like it needed forty-seven questions needed six.

That a learner's time is the scarcest resource in the system, and every design argument eventually reduces to it: six questions instead of forty-seven, a grade band instead of a node, one lesson for half a class instead of twenty-four conversations.

And that a claim nobody checks is a claim that quietly stops being true. Every load-bearing statement in this repo now has a script or a test that fails if it becomes false — several of them were false when we first went looking.

What's next for Strata

Answering in the student's language. A question typed in Vietnamese is classified and matched correctly today — we tested it — but the probes come back in English. Detecting the language and authoring in it is the next thing worth building, because the students most likely to be carrying a five-year-old gap are not the ones studying in their first language.

Beyond that: measuring whether a repair held. A check question at the end of a step proves the idea landed for ninety seconds. Coming back a week later and re-probing the same node would prove something worth knowing.