Faultline

See the thinking behind every mistake

Comment

Inspiration

Faultline started with a problem that almost every teacher has faced.

A student can seem to understand a lesson in class, answer correctly with help, and still fail when they have to solve the same type of problem alone.

When that happens, teachers usually see the final answer and the score. They know who got the question wrong, but they often do not know what the student was actually thinking.

That matters because the same wrong answer can come from completely different causes.

One student may have misunderstood the concept. Another may understand the concept but follow the wrong procedure. A third may know the calculation but struggle to understand the wording of the question.

A normal grading system treats all three students as simply wrong.

But these students do not need the same help.

One may need the concept explained again. One may need a wrong habit corrected. One may need support with reading and interpretation. If the teacher cannot see the difference, the safest option is often to reteach the entire class.

That takes time, frustrates students who already understand, and may still fail to help the students who need something more specific.

We wanted to build a tool that could look beyond the score and help teachers understand the process behind the mistake.

That idea became Faultline.

Faultline treats student work as evidence. It tries to uncover the hidden procedure that produced each answer and turns that understanding into a clear next step for the teacher.

What it does

Faultline analyzes student work and identifies the most likely procedure behind the answers.

It does not stop at saying:

“This student got four questions wrong.”

Instead, it can explain:

“This student is adding the numerators and denominators separately.”

That is much more useful because it shows the teacher what the student is actually doing.

Faultline compares the student’s responses against a library of executable procedures. Each procedure works like a small program. It receives a maths problem and produces the answer that a student following that method would likely write.

The system then checks which procedure best explains the full pattern of work across several questions.

Faultline shows the teacher:

  • The most likely procedure
  • Other possible explanations
  • The evidence behind the result
  • The confidence of the diagnosis
  • The best next questions to ask
  • A recommended teaching action

The system does not force a diagnosis when the evidence is weak.

If several explanations are still possible, Faultline selects the questions that would separate them most effectively. If the handwriting is unclear or the responses are inconsistent, it says that more evidence is needed.

One of Faultline’s strongest features is prediction before reveal.

The system studies the student’s completed work and predicts the answer that the inferred procedure should produce on a hidden problem.

Only after the prediction is committed does Faultline reveal the student’s actual answer.

If the two answers match, it shows that the system has identified a procedure that can explain new work, not just describe mistakes that have already happened.

Finally, Faultline turns the results into an action plan.

The teacher can see which students need a conceptual explanation, which students need targeted practice, and which students need a closer look before any decision is made.

How we built it

We built Faultline as a full-stack application with a Python and FastAPI backend and a lightweight web interface designed for teachers.

The central part of Faultline is a neuro-symbolic diagnostic system.

The neural part helps read student work and propose possible reasoning patterns.

The symbolic part represents those patterns as executable procedures. These are not vague labels or generated explanations. They are rules that can be run against different maths problems to produce exact predicted answers.

Faultline then uses a deterministic Bayesian inference engine to compare the procedures.

For every possible procedure, the engine asks:

How likely is it that this procedure would produce the student’s observed answers and intermediate steps?

It combines the prior probability of each procedure with the evidence from the student’s work and calculates a posterior probability.

This allows Faultline to show a distribution of possible explanations instead of pretending that one answer is always certain.

The system also accounts for recognition uncertainty. If a handwritten answer may have been read incorrectly, that uncertainty reduces the strength of the diagnosis.

When two procedures remain close, Faultline uses expected information gain to choose the next question.

It looks for a problem where the competing procedures predict different answers. That question provides the teacher with the most useful new evidence.

We also designed a verifier-gated process for new mistake patterns.

If the known procedure library cannot explain a student’s work, an AI model may propose a new symbolic hypothesis in a restricted format.

The AI does not get to declare the diagnosis.

The proposed rule must be executed against the student’s actual responses. It is accepted only if it reproduces the observed pattern. If it fails, it is rejected.

We also took special care with the prediction-before-reveal feature.

The hidden answer is never sent to the browser with the initial data. The prediction is created on the server, and the real answer is protected behind a signed, time-limited proof token.

This prevents the demo from becoming a visual trick where the answer is already available in the page source.

The project also includes strict upload validation, confidence gating, rate limiting, security headers, structured logging, Docker support, evaluation tools, automated tests, and release verification scripts.

Challenges we ran into

One of the biggest challenges was deciding what Faultline should honestly claim.

It would have been easy to show impressive numbers such as predicted mastery, future improvement, or expected learning gains.

But we did not have the classroom data required to support those claims.

We decided not to invent them.

Instead, we focused on something we could test directly. If Faultline claims that a student is following a procedure, that procedure should predict the student’s answer on a new problem.

Another challenge was that wrong answers are often ambiguous.

A single answer may fit several possible procedures. We needed to reason across multiple questions, use intermediate steps when available, and account for uncertainty in the transcription.

Building the procedure library was also difficult.

Each procedure had to be understandable to a teacher, general enough to work on new problems, and precise enough for the computer to execute.

We also faced an important security issue during development.

In an earlier version, the hidden answer used in the prediction demo was present in the browser data, even though it was not visible on the screen.

A technical judge could have inspected the page and found it.

We rebuilt the feature so the answer remains on the server until the prediction is committed and a valid proof token is checked.

This made the feature technically honest and much stronger.

Another challenge was keeping the AI useful without giving it too much authority.

An AI model can be helpful for reading handwriting and proposing a possible new pattern, but it should not be trusted to label a student without verification.

That is why Faultline uses AI for perception and hypothesis generation, while the deterministic engine handles execution, evidence scoring, uncertainty, and final acceptance.

Accomplishments that we're proud of

We are proud that Faultline does not treat every wrong answer as the same kind of failure.

It can distinguish between a consistent incorrect procedure, inconsistent responses, uncertain recognition, and cases where more evidence is needed.

We are also proud of the prediction-before-reveal system.

It gives teachers and judges a direct way to test whether the diagnosis is meaningful.

Instead of asking people to trust a generated explanation, Faultline makes a prediction that can be proven right or wrong.

Another accomplishment is the way Faultline handles uncertainty.

The system can refuse to make a diagnosis when the evidence is not strong enough.

In education, an incorrect label can lead to the wrong intervention. Knowing when not to answer is part of being responsible.

We are also proud of the technical discipline behind the project.

The corrected system includes:

  • Executable procedure models
  • Bayesian inference
  • Information gain
  • Recognition uncertainty
  • Signed proof tokens
  • Secure upload handling
  • Strict validation
  • Rate limiting
  • Automated regression testing
  • Release integrity checks

Our controlled regression dataset ranks the assigned procedure first for all twelve demonstration students.

Eleven students receive a named diagnosis, while one low-confidence case is deliberately withheld.

These are synthetic regression results, not claims of classroom accuracy, but they show that the reasoning engine behaves consistently and can be tested.

What we learned

We learned that the most useful education technology does not simply generate more information.

It helps a teacher make a better decision.

A detailed AI explanation is not valuable if it does not change what the teacher should do next.

We also learned that the same mistake can carry very different meanings.

Looking only at correctness removes the most useful part of the evidence, which is the procedure that produced the answer.

We learned that neural models and deterministic reasoning work better together than either approach alone for this problem.

The neural model is useful for reading messy student work and proposing possibilities.

The symbolic and Bayesian engine is useful for testing those possibilities, measuring uncertainty, and explaining why one procedure fits better than another.

We also learned that confidence should not be hidden.

Many systems return one answer because the interface expects one answer. Faultline became more trustworthy when it was allowed to say:

“We do not know yet.”

Finally, we learned that honesty can make a project more technically impressive.

By removing unsupported claims and making the core diagnosis falsifiable, Faultline became easier to explain, easier to evaluate, and harder to dismiss as another generic AI education tool.

What's next for Faultline

The next major step is testing Faultline with real handwritten student work.

We want to evaluate three parts separately:

  1. How accurately the system reads the student’s answers and intermediate steps
  2. How accurately the Bayesian engine identifies the correct procedure after the transcription is reviewed
  3. How accurately the full system works from worksheet image to final recommendation

We also plan to expand the procedure library beyond fraction addition.

Future versions could support decimals, percentages, ratios, equations, algebraic manipulation, and other areas where students often follow consistent but incorrect procedures.

We want to make teacher review faster and more flexible.

Teachers should be able to correct the transcription, approve or reject evidence, and see how those corrections change the posterior diagnosis.

We also plan to add longitudinal tracking.

This would allow teachers to see whether a procedure disappears after an intervention, continues over time, or returns in a different topic.

Another important step is classroom evaluation with teachers.

We want to study whether Faultline helps teachers create more targeted lesson plans, reduces unnecessary whole-class reteaching, and improves the quality of follow-up questions.

We will not claim those outcomes until they are measured.

The long-term goal is not to replace teachers, grading, or professional judgment.

The goal is to give teachers better evidence.

A wrong answer is not only a result.

It is a trace of the reasoning that produced it.

Faultline helps teachers follow that trace.

Built With

Share this project:

Updates

Submission history