Faultline

Inspiration

When I was younger I got the basis of addition of fractions worng, which carried over to my later years and really hindered my test scores and it took me months to finally find the problem and solve it, which faultline solves easily

What it does

Faultline works out why a student got something wrong. Every wrong option is engineered to be the exact output of one specific broken rule, so the answer they pick identifies the rule they're running. It tracks a live probability over 19 named misconceptions, and when a fault keeps recurring it stops drilling and prescribes the targeted repair instead.

How we built it

I started with the content, because the content is the hard part. Nineteen misconceptions pulled from the maths-education literature: whole-number bias in fractions, the operational view of the equals sign, over-generalised distribution. Each one carries the faulty rule written in the student's own voice, a symbolic signature, and a repair strategy a teacher would actually use.

Then the item bank. Forty-three questions where every wrong option is the literal output of one of those broken rules. Answering 1/2 + 1/3 with 2/5 isn't a near miss. It's what adding across the bar produces, and nothing else produces it. That property is the whole trick, and it can't be automated. Someone has to sit down and author it.

The engine runs two Bayesian models side by side. Bayesian Knowledge Tracing tracks whether you can do a skill. A separate belief updater tracks which broken rules you're running, and it updates every misconception an item baited, not just the one that fired, so a handful of questions can clear a wide area of the map. A scheduler sits on top and picks the next question by expected information gain instead of difficulty.

That first version worked. It was also a bit of a lie, because every parameter in it was a number I had guessed. So I went and got real data.

ASSISTments: 6.1 million student responses. I fit the BKT parameters properly with Baum-Welch, held out 30% of the students, and checked whether the fitted model predicted their answers better than my guesses did. It did, 0.676 AUC against 0.641, winning on 17 of 20 skills. All four of my original constants turned out to be wrong, and wrong in directions I wouldn't have predicted.

Then I tried to go further and failed twice, which ended up being the most useful part of the build.

The first attempt was to discover the misconceptions themselves, unsupervised, from 16 million Eedi answers. A latent-class model over wrong-answer choices. It fit beautifully in-sample and then lost to a plain item-difficulty baseline on held-out data. The second attempt was to model students as aptitude profiles rather than concept checklists, on the theory that a spatial thinker does better in geometry than in arithmetic. Held-out AUC came back identical to a single overall-ability score. 0.7869 either way. The positive manifold ate it.

Both nulls are in the repo. They're the reason the taxonomy stayed expert-written instead of learned, and I'd rather show the failed experiments than quietly drop them.

The last piece was the stopping rule. Once a fault is confirmed and the student has produced it three times without repairing, the engine stops serving that fault entirely and prescribes the specific fix instead. Getting that right took one more fix than expected: the first version stopped drilling permanently, which meant it could never observe the student improving, and faults froze on the map forever. Evidence has to be allowed to go stale.

Challenges we ran into

Two experiments failed outright. Discovering misconceptions unsupervised from 16 million answers looked flawless in-sample, then lost to a plain difficulty baseline on held-out data. The model had memorised, and only the held-out check caught it. Modelling students as aptitude profiles died the same way, with identical AUC to a single ability score. The subtler problem was engineering. My first stopping rule stopped drilling a confirmed fault permanently, which meant the engine could never watch a student improve. Faults froze on the map forever.

Accomplishments that we're proud of

The engine finds misconceptions it was never told about. Across 240 simulated students it recovers 78% of them at 96% precision, and a student who holds none is never falsely accused of one. That last number matters more to me than the other two. Telling a kid they have a systematic error they don't actually have is the worst thing this system could do, so I bought precision at the cost of recall on purpose.

The trained skill model also beat my own hand-set parameters on 394,000 held-out responses, winning on 17 of 20 skills. Every constant I'd guessed turned out to be wrong.

And when a held-out check caught a model that had passed every internal test I'd written, I threw out the model instead of the check.

What we learned

Every constant I guessed was wrong once I checked it against six million real responses, and wrong in directions I wouldn't have predicted. In-sample metrics lie. The misconception model passed every internal check I'd written and still lost out-of-sample. The bigger lesson was about personalisation itself: twice I went looking for per-student structure in the data, and twice item difficulty and overall ability swamped it. There's a real ceiling there that the marketing around adaptive learning doesn't mention. Knowing roughly where it sits is what made an expert-authored taxonomy the right call instead of a fallback.

What's next for Faultline

The method needs subjects where wrong answers come from mechanical, nameable rules. Physics, chemistry, grammar and music theory all qualify. History and essay-writing don't, and I'd rather say that than overclaim. Nearer term: widen the taxonomy past nineteen misconceptions, since anything outside it currently registers as an unmapped error and changes no beliefs. Then validate against real students instead of simulated ones. The stopping rule is what I'd most want a teacher to test, because whether the prescribed repair actually clears the fault is the one question my data can't answer.

Built With

• claude
• duckdb
• next.js-16
• node:sqlite
• numpy
• python
• scikit-learn
• scipy
• tailwind-4
• typescript