AceSAT — Adaptive Multi-Agent SAT Tutor

Not a chatbot — an AI agent that plans, adapts, and tutors each student toward a better SAT score, built for students who've never had a real tutor.

Comment

Inspiration

Students in well-resourced school districts get private SAT tutors who notice exactly what they're struggling with and adjust on the fly. Students in underserved public schools usually get a static practice book: the same questions, in the same order, regardless of where they're actually stuck. That's not a content-access problem; free SAT questions exist everywhere. It's a personalization problem. And continuous, adaptive attention to one student's actual weak spots is exactly the kind of thing AI is good at, and exactly the kind of thing that's unaffordable at scale for the students who need it most. I wanted to build the thing a real tutor does: diagnose, plan, adapt, remember, not a chatbot wearing a tutor costume.

What it does

AceSAT is an adaptive, multi-agent SAT tutor. A student takes a short diagnostic (they choose 10 or 15 questions), which produces a real evidence-based mastery estimate per topic instead of a flat guess. From there, the system adaptively routes between Math, Reading, and Language, targeting weak topics first, respecting prerequisites (it won't push Geometry before Algebra is solid), and occasionally reviewing a topic that's gone stale so it doesn't quietly get forgotten. Every answer updates a real Bayesian Knowledge Tracing model, difficulty scales across four tiers as mastery climbs, and questions are validated before they ever reach the student, rejecting duplicate answer choices, repeated questions, and even math questions that are symbolically unsolvable. At the end of a session, a separate reflection agent narrates what actually happened, and a dashboard shows mastery, a predicted SAT score, and session history over time.

How I built it

The backend is Python/FastAPI with SQLite, structured as a genuine multi-agent pipeline rather than one LLM call in a loop:

  • Planner Agent: sets one session-level goal (e.g. "focus on Algebra, it's weakest; also review Reading, it's gone stale")
  • Orchestrator: executes that goal turn by turn, routing between subjects with enough flexibility to react to real-time performance shifts
  • Specialist agents: pick the next topic within a subject using an epsilon-greedy bandit
  • Knowledge Graph: a DAG of topic prerequisites the bandit has to respect
  • Bayesian Knowledge Tracing: real probabilistic mastery updates after every answer, plus a forgetting-curve decay for untouched topics
  • Question Generator: an LLM call wrapped in a validation layer (duplicate detection via TF-IDF similarity, symbolic math verification via sympy, self-contradiction checks) with a hand-verified static fallback if live generation fails twice
  • Reflection Agent: a second, separate LLM call that narrates session progress, with its own deterministic fallback

The frontend is a single-page HTML/CSS/JS app with no framework, deliberately kept lightweight so it stays usable on modest devices and slower connections. It's deployed as two independent pieces: the backend on Azure App Service, the frontend on Netlify, so the demo link is a real, live deployment, not a local-only build.

Challenges I ran into

A few were worth calling out specifically, because they taught me the most.

Routing that got stuck. The session Planner sets one primary-focus subject at the start of a session, and the Orchestrator is supposed to revisit that choice if it stops making sense. I found a case where it didn't: the primary subject sat well below "mastered," while a different subject had genuinely become weaker, but nothing re-evaluated the plan, because the only trigger for an early refresh was the primary subject crossing a near-mastery threshold. The Orchestrator kept routing to the no-longer-weakest subject regardless. The fix was removing that mastery-threshold gate entirely: the plan now refreshes whenever the actual weakest subject diverges meaningfully from the current focus, not just when the current focus has been "solved."

Silent generation failures. I found a case where a generated question's fourth answer choice wasn't a clean option at all. It was the model leaking its own self-correction into the choice text itself ("...is incorrect, the correct answer is..."), something no existing check caught, because duplicate-answer detection only flags identical text and my explanation-consistency check only ever looked at the explanation field, never the choices themselves. That pushed me to add a targeted check on choice values directly, on top of the existing sympy symbolic verification and self-contradiction detection for explanations.

I also hit the much more mundane but very real "missing file" class of bug: a FastAPI backend that imports a dozen local modules will crash at startup if even one is missing locally, and every symptom downstream (login, dashboard, question loading) looks identical: nothing works. That taught me to be explicit in my own README about exactly which files are required, rather than assuming it's obvious.

Accomplishments that I'm proud of

Building validation into the question pipeline instead of trusting the LLM by default: duplicate detection, symbolic math verification, self-contradiction and leaked-reasoning checks, and a graceful static fallback that means a generation failure never breaks the session for the student. Building a real evaluation harness that reports routing accuracy and generation-quality metrics from logged data, not just a demo that looked fine when I happened to be watching it. And backing all of it with an actual automated test suite: 33 tests covering routing correctness, mastery consistency between the live session and the dashboard, and question-validation behavior, several of which I deliberately verified fail against the earlier, buggy version of the code, so I know they're catching something real rather than just passing by default.

I also added rate limiting on the generation and auth endpoints, and structured JSON logging throughout, since this needed to survive being deployed and actually used, not just demoed once.

What I learned

The gap between "an LLM answers questions" and "an agent that adapts, plans, and remembers" is mostly in the boring parts: validation, staleness detection, graceful degradation, and being honest about what's a real model (Bayesian Knowledge Tracing) versus an explainable heuristic (the score predictor, the diagnostic prior). I think that honesty is a feature, not a weakness: a system that's clear about its own limitations is more trustworthy than one that fakes precision it doesn't have.

What's next

Real usage data to properly calibrate the heuristic pieces, a wider question bank, migrating off SQLite to something that survives redeploys cleanly, and extending the same multi-agent approach beyond the SAT to other subjects where the same core problem, personalized attention that doesn't scale for underserved students, applies just as much.

Built With

Share this project:

Updates

Submission history