• 

  Error Archaeologist Logo

GPT Error Archaeologist

Error Archaeologist does not guess what is in a student’s head. It proposes evidence-linked explanations, then generates a mathematically verified question that helps distinguish between them.

About the project

Error Archaeologist is a working technical demo built for the OpenAI Build Week Education category. It explores a narrow but important gap in math education: grading shows that an answer is wrong, while tutoring provides remediation, but teachers must still determine why a particular mistake happened.

One incorrect answer can have several explanations. A student may hold a repeatable misconception, make a one-time slip, copy something incorrectly, or write an ambiguous symbol. Error Archaeologist treats those explanations as hypotheses—not facts about the student’s mental state.

GPT-5.6 Luna examines a curated synthetic handwritten solution and returns two evidence-linked candidate explanations. The system then creates a follow-up problem whose predicted answers differ between candidates. SymPy independently verifies those predictions before the question appears. The student’s response changes hypothesis support; ambiguous evidence can cause the system to abstain.

NAEP 2024 reports that 39% of US eighth graders scored below Basic in mathematics. This establishes problem scale, not demand for this product or evidence that it improves learning. Current demo proves an end-to-end technical interaction, not classroom efficacy.

Inspiration

Teachers often see what went wrong without enough evidence to know why.

Consider:

-3(x - 2) = -3x - 6

This could reflect incorrect handling of a negative sign, distribution to only one term, or a simple transcription slip. Each explanation suggests a different teaching response.

I did not want to build another AI grader that converts uncertain evidence into a confident label. I wanted to explore a more honest workflow: propose plausible explanations, show where each explanation comes from, and ask one small question that can produce new evidence.

That makes diagnosis falsifiable. The AI does not get the final word—the student’s next answer can support, weaken, or fail to distinguish its hypotheses.

What it does

1. A teacher selects one of three curated synthetic handwriting samples.
2. GPT-5.6 Luna analyzes the mathematical steps through the OpenAI Responses API and Structured Outputs.
3. The interface shows the observed transition and two candidate explanations, each connected to evidence from the work.
4. The model proposes a differentiating follow-up problem and predicts how each candidate would answer.
5. SymPy independently verifies that candidate predictions are mathematically distinct and do not equal the correct answer.
6. The teacher can use either displayed prediction as a simulated student response.
7. The response updates candidate support without claiming proof of the student’s mental state.
8. Ambiguous input, invalid probe mathematics, or an unmatched response can produce abstention instead of a forced diagnosis.

Current demo intentionally supports curated synthetic samples only. It does not yet provide uploads, authentication, teacher confirmation, class aggregation, or real-student data processing.

How I built it

The application is a modular monolith packaged as one Docker image:

• Frontend: React and Vite
• Backend: FastAPI with three JSON endpoints
• Model: GPT-5.6 Luna with medium reasoning
• Model interface: OpenAI Responses API with Structured Outputs
• Independent verification: SymPy with restricted expression parsing
• Demo persistence: disposable SQLite state
• Deployment: one public Google Cloud Run service

GPT-5.6 performs multimodal interpretation, evidence localization, hypothesis generation, and probe proposal. It does not certify its own mathematics. SymPy separately checks the probe predictions before the backend marks a probe as verified.

Codex served as an agentic development collaborator throughout the project. It helped audit research claims, reduce scope, design architecture, implement the FastAPI and React flows, build tests, debug integration issues, package Docker deployment, and prepare Cloud Run configuration. I made the product decisions: choosing teachers and tutors as initial users, rejecting mind-reading language, requiring independent mathematical verification, adding abstention, and cutting infrastructure that did not improve the judge-visible demo.

Automated verification currently includes 15 backend tests, four frontend workflow tests, and a production frontend build. Tests use a fake model adapter, so judges can reproduce core behavior without consuming an API key.

Challenges I ran into

Preventing confident guesses

A language model can turn weak evidence into a polished diagnosis. I designed the output around candidate hypotheses and evidence updates instead of final labels. Ambiguous samples have an explicit abstention path.

Verifying the differentiating question

Generating another algebra problem is easy. Generating one where competing hypotheses predict meaningfully different answers is harder. The backend uses SymPy to reject probes whose predictions collide or accidentally equal the correct answer.

Keeping model and verifier separate

Letting GPT-5.6 generate and approve its own mathematics would provide little assurance. I kept deterministic verification outside the model response and restricted the expression grammar accepted by SymPy.

Cutting scope

The initial architecture included uploads, durable PostgreSQL storage, teacher review, class heatmaps, and broader evaluation. Those features would have produced more surface area but a weaker hackathon demo. I reduced the project to one complete loop that judges can test.

Staying honest about evidence

Three synthetic examples and passing tests do not prove diagnostic accuracy, educator demand, or improved learning. Public research supports the importance of the problem; product validation still requires external artifacts and educator studies.

Accomplishments that I’m proud of

• Shipped a public end-to-end demo rather than only an architecture proposal.
• Connected every candidate explanation to visible evidence.
• Built a follow-up question whose candidate predictions are independently checked with symbolic algebra.
• Made new evidence change hypothesis support instead of displaying a static AI verdict.
• Included abstention as core product behavior.
• Separated GPT-5.6 reasoning from deterministic mathematical verification.
• Preserved a reproducible fake-model path requiring no API secret or budget.
• Kept research, current implementation, and future product claims clearly separated.

What I learned

Diagnosis is not classification from one observation. It is a sequence: propose explanations, identify what evidence would distinguish them, collect that evidence, and update support.

I also learned that deterministic software becomes more valuable—not less—when placed around a capable model. GPT-5.6 handles interpretation and hypothesis generation; SymPy enforces a mathematical invariant the model cannot waive.

Most importantly, public statistics can establish that a problem matters, but they cannot establish that a specific product works. Building from Taiwan did not prevent me from researching the US education system through public government and academic sources. However, real educator interviews, representative evaluation, workflow observation, and pilot behavior must complete that evidence chain.

What’s next

Next steps focus on validation before expansion:

• Build a sealed external evaluation set containing repeatable misconceptions, slips, ambiguous notation, and correct-answer-but-invalid-process cases.
• Measure hypothesis quality, evidence localization, abstention behavior, probe validity, latency, and cost.
• Interview Grade 7–9 intervention teachers and tutors about workflow fit and false-positive tolerance.
• Add privacy-reviewed, de-identified uploads only after reliability and data-handling requirements are defined.
• Add teacher accept, revise, and reject controls before any result enters an aggregate.
• Replace disposable SQLite with durable authenticated storage.
• Develop a sourced, versioned misconception taxonomy for one algebra unit.
• Test a paid pilot with intervention or tutoring providers before claiming pricing or market demand.

Built With

• cloudrun
• docker
• openai
• python
• react
• sqlite
• typescript
• vite