EduAgent — Collaborative Partner Socratic Mentor

EduAgent project cover image.
Overall system architecture diagram
Teacher dashboard displaying class-wide learning results
Automated email report draft generated for the teacher
Live Socratic debate interface between the student and AI

Inspiration

A teacher with 40+ students cannot give each one detailed feedback on their reasoning, only on their grammar. Most AI writing tools make that worse in a different way: they write the better sentence for the student. We wanted the opposite, a tool that makes the student do the thinking, and gets out of the way the moment it's tempted to do that thinking for them.

If the assistant rewrites the draft, patches the logic gap, smooths the claim, the student learns to copy the model's output, not how to build an argument. So the rule we built around is simple: use the model to teach reasoning, not to replace it. The result is a Socratic partner that argues back, remembers where a specific student keeps getting stuck across assignments, and hands the teacher something they can act on without reading forty essays first.

What it does

EduAgent runs as two tiers that don't know about each other's internals, connected by one event.

Tier 1: Per-Student Adaptive Socratic Pipeline

Messy input, on purpose. A student can paste text, share a Google Doc link, or photograph a handwritten page. Photos go through Gemini Vision OCR, transcribed verbatim, misspellings and cross-outs kept, not cleaned up. Because a model can misread the same stroke the same way twice and still sound confident about it, we run a second transcription with a different model family, Gemma 4 (gemma-4-26b-a4b-it-maas), and diff the two outputs with difflib. If they disagree past a threshold, the essay routes to a human queue instead of getting scored on a guess.
Persona routing, and the routing is shown, not hidden. The agent reads the student's history and the current essay across four dimensions, then picks one of four Socratic personas, the Skeptic for unsourced claims, the Devil's Advocate for one-sided arguments, the Nitpicker for internal fallacies, the Expander for overgeneralization, and shows the student why that one was picked, with a manual override if they want to practice a different weakness.
A 3-turn debate, not a chat. The persona holds its role across all three turns using prompt-level anchoring, so it doesn't drift back into being a generic helpful assistant halfway through.
A regex validator that never calls an LLM. Every question the model generates gets checked by plain Python before the student sees it, did it leak the answer, ask two questions at once, run too long? If so, it's regenerated or replaced with a fallback question. This check costs nothing and can't hallucinate.
A radar chart, then a rewrite. After the debate, the student sees their scores across four axes (logical coherence, evidence quality, scope awareness, counterargument handling) as an SVG radar chart, then rewrites their core claim to address what the debate exposed, with feedback on the rewrite itself.

Tier 2: Class-Wide Aggregator & Teacher Co-Pilot

When an essay finishes scoring, Tier 1 publishes one event to Pub/Sub. A Cloud Run subscriber picks it up, checks it hasn't already been processed, and runs:

Fallacy clustering across the class. Scans every active student profile in the class looking for the same reasoning error showing up in more than one student.
A priority ranking that's arithmetic, not a prompt. $$Priority = 3.0 \cdot \text{stuck_streak} + 2.5 \cdot \text{score_decline} + 1.0 \cdot \text{inactivity_days} + 1.5 \cdot \text{shared_fallacy_weight}$$ The weights live in config.PRIORITY_WEIGHTS. A teacher (or a judge) can read the exact number that put a student at the top, nothing is buried in a model's summary.
Gemini writes the digest. It takes the ranking and the fallacy clusters and drafts a classroom report with a concrete 15-minute mini-lesson suggestion attached.
The teacher still has to click Send. The system drafts the email in the teacher's own Gmail account, it does not send it, and logs a row to a Google Sheet the teacher configures (with URL parsing, multi-tab fallback, and a connection-test button, because the first version of this broke silently on a renamed sheet).

How we built it

Google ADK2: the pipeline is a Workflow of FunctionNode units, with ctx.route branching typed-text input away from the image/OCR path.
Gemini 3.5 & 3.7 Flash, via Vertex AI with a dedicated service account. 3.5 handles the fast per-turn work, intake, debate, summarization. 3.7 handles the heavier class-digest synthesis, where latency matters less than quality.
Firestore (Native Mode) holds student profiles and debate sessions, updated with atomic transactions so two events for the same student can't race each other into an inconsistent profile.
Gemma 4 (gemma-4-26b-a4b-it-maas) is the second, independent model in the OCR consistency check (ADR-028). It runs as MaaS on the same Vertex AI global endpoint and credentials as Gemini, so there's no extra infrastructure to stand up. The shared-capacity endpoint returns 429 request queue is full often enough that we saw it in 4 of 10 calls during early testing, retried, then falls back to a second same-model pass, with the digest recording which model actually produced the text used.
Pub/Sub is the only thing that connects Tier 1 to Tier 2. Neither tier calls the other directly.
Cloud Run hosts the FastAPI app in a multi-stage, non-root container, handling authenticated Pub/Sub push deliveries.
Cloud Trace + OpenTelemetry wrap every graph node. The batch pipeline traces one essay end-to-end in a single trace (8 nested spans, intake through profile_mutator); the interactive web API traces per-request (e.g. /api/debate/start's sanitizer → summarizer → persona_selector → debate_loop), since a live debate spans several separate HTTP calls rather than one continuous process.
An eval suite with 50 deterministic cases across four layers (safety, behavioral discipline, long-term memory, learning outcomes), no LLM judges anywhere. Every one of the 50 is checked with sabotage testing: break the code on purpose, confirm the test goes red. Twelve originally couldn't (see below).

Challenges we ran into

Gmail's "compose-only" scope isn't compose-only. We wanted the Gmail integration locked to draft creation at the OAuth layer, but gmail.compose officially permits sending too, Google doesn't offer a narrower scope. Since we couldn't enforce this with a permission, we enforced it with a test: an AST parser scans the integration module on every build and fails if .send() is called anywhere in it.
OCR hallucinated on blurry photos, and our first fix couldn't have caught it. On bad photos, Gemini Vision would occasionally invent whole sentences while reporting "high" confidence on its own output. Our first attempt: transcribe twice with Gemini, diff the two strings, downgrade confidence if they disagree. It looked reasonable until we noticed the flaw, if the same model misreads the same stroke twice, both passes agree on the same wrong text, and the diff sees perfect agreement. We swapped the second pass for Gemma 4, a different model family with uncorrelated errors. On one real sample, two Gemini passes scored 0.781 similarity, passing our old check, while Gemini against Gemma scored 0.294. The cross-model version caught an essay that was, in fact, unreadable.
The old similarity threshold didn't survive the fix. Two different model families format the same handwriting differently, punctuation, line breaks, where each one gives up and writes [[unclear]], so applying our old 0.75 same-model cutoff to a cross-model comparison started rejecting legible essays. We measured it properly on our 12 real samples: legible pages score 0.989–1.000 within one model family but only 0.729–0.998 across two families, and unreadable pages cluster at 0.265–0.294 regardless. The cross-model threshold is now its own constant, 0.50, sitting in the gap between those two clusters.
/healthz returns 404 on Cloud Run, and it's not our bug. Knative's proxy layer intercepts that exact path before it reaches the app. We moved the health check to /health-check.
We audited our own eval suite and found 12 tests that could never fail. Eight of Layer 4's "cognitive growth" tests were literally assert 8 - 2 >= 4 against fixture constants, they'd pass with the entire src/ directory deleted. A separate group of persona tests built the prompt string inside the test itself and then asserted the persona name was present in the string they'd just built. Both are the same failure mode: an assertion checking its own setup. We rewrote them to call the real functions (build_system_instruction(), merge_reflection_into_profile()) and real scorer output. Now, disabling persona anchoring fails 4 of 4 of those tests; deleting the scorer artifact fails the other 4 of 4.
Our headline learning-outcome number was fake, and we'd written it ourselves. The script behind our original +5.62 claim had before_scores and after_scores hardcoded as literals, no essay was ever scored to produce them. We rewrote it to run both essay versions through the actual summarize_essay() → score_essay() path on Vertex AI, scoring one text at a time with the scorer never told which one was the revision. The real number is +2.75, across 7 of 8 scenarios, one didn't improve. Reporting half our original number was the only honest option once we found the bug.
The session secret protecting teacher tokens had a checked-in default. EDUAGENT_SESSION_SECRET fell back to a value committed in the repo, and nothing forced it to be overridden at deploy time, anyone reading the source could forge a role=teacher token for any class. Fixed by moving the key to Secret Manager and making the container refuse to start on Cloud Run if the default is still active. The same audit also found that all five student-facing debate endpoints had no authentication at all, while every teacher-facing route did, so any caller could write into any student's profile undetected. Those five endpoints now require a token scoped to that specific student.
Fixing one leaked credential, we missed two more of the same kind. After moving the session key to Secret Manager, a later review found our deploy script was still passing the Gmail and Sheets OAuth refresh tokens through --env-vars-file. Cloud Run stores plain env vars in the revision spec in cleartext, so anyone with run.services.get, a read permission, not an admin one, could pull both refresh tokens straight out of gcloud run services describe. All three credentials now mount via --update-secrets instead, and we added an AST test that fails the build if a credential is ever inlined again, plus a doctor.py check that flags any cleartext credential on the live revision.

Accomplishments that we're proud of

Safety checks that don't touch the model. The sanitizer and challenge validator are plain regex and string matching, they catch injection attempts and answer leaks instantly, with no token cost and nothing to hallucinate.
An event boundary that actually decouples. A spike in student submissions can't block the student-facing UI or stall the teacher digest, because Pub/Sub sits between the two tiers, not a direct function call.
A real handwriting benchmark, not synthetic text. 12 photos of actual handwritten essays, cursive, cross-outs, bad lighting, tilted framing, used to validate the OCR path, not generated sample data.
A priority list a teacher can argue with. Because the ranking is arithmetic against config.PRIORITY_WEIGHTS, a teacher can ask "why is this student first" and get a real answer, not an LLM's paraphrase of one.
Numbers we checked before we believed them. The memory A/B test (3 essays, one profile) showed 0% repeated interventions with persistent memory versus 1 repeat without it. The learning-outcome measurement ran 8 author-written before/after pairs through the live production scorer, blind to which version was the revision, for a mean +2.75 point gain on the targeted axis across 7 of 8 cases. n=8 pairs, not 8 students, no control group, it measures whether the scorer detects the improvement each persona is meant to produce, not classroom-wide learning gains.

What we learned

LLMs are good at generating, bad at gatekeeping. Anything that has to reliably block a failure mode, leaked answers, malformed output, unsafe actions, needs to be a hard code check: regex, an AST parser, a schema. Not another model call.
Put tracing in early. OpenTelemetry across every ADK2 node meant that when a request took 20+ seconds, we could see exactly which node ate the time instead of guessing.
The platform doesn't behave like the docs say it does. Cloud Run's health-check path collision and Pub/Sub's IAM quirks only showed up once we tested against the real services, a mock would have hidden both.
A green test suite proves nothing by itself. The right question isn't "did it pass" but "was it able to fail", sabotage testing is how we found the 12 dead assertions in our own suite, and we wouldn't have found them any other way.
Read the code against the docs, not the other way around. Every serious bug in our final audit pass came from taking a specific claim in our own documentation and going to find the line of code that was supposed to implement it. Twice, that line didn't exist.
A fix is usually narrower than the bug. We patched one leaked secret and missed two more of the same shape; we authenticated the teacher's routes and left the student's routes open. Both times, the second half of the problem was the same category of mistake as the first, just a different endpoint or a different credential. The fix that actually holds isn't "be more careful next time," it's a test or a doctor.py check that catches the whole category, not just the instance we happened to notice.

Other data sources used

No third-party or public datasets were used. Everything below was produced by the author during the Submission Period:

12 photographs of real handwritten essays (eval/test_images/), shot on ordinary paper, deliberately including cursive, pencil, cross-outs, faded ink, uneven lighting, and tilted framing. Two are genuinely illegible on purpose, to check that the OCR cross-check routes them to pending_essays instead of scoring garbage.
8 author-written thesis pairs (eval/results/learning_outcome_measured.json), a before/after pair per targeted cognitive weakness, used as the learning-outcome benchmark. n=8 pairs, not 8 students, no control group.
Synthetic demo student profiles (scripts/seed_student_profiles.py), five fictional trajectories (improving, declining, stuck, inactive, shared-fallacy) that populate the teacher dashboard for the demo. No real student data anywhere in this project.
Sample typed essays (assets/sample_essays/), author-written, for the typed-text path.

No student, classroom, or personally identifiable data was collected, purchased, or scraped.

Mandatory Disclosure

This project is inspired by the author's prior hackathon entry, CritiqAI (Kaggle/Google AI Agents Intensive Vibe Coding Capstone, Education track). All code, prompts, schemas, workflows, and evaluation suites in this repository were written completely from scratch during this Submission Period. The prior project was a reference for what worked and what didn't architecturally, none of its source was forked, copied, or incorporated, and it's excluded from this repository's git history by .gitignore. Track: Collaborative Partner.

What's next for EduAgent

Close the loop on interventions. Right now we surface a priority list; we don't yet track whether a teacher's chosen intervention actually helped that student on the next assignment (Observe → Diagnose → Intervene → Measure → Adapt).
Trends across a term, not just one essay. Detect whether a class-wide reasoning weakness is fading or persisting across multiple assignments.
Harden it for more than one class. Formal multi-school tenancy isolation, data retention windows, accessibility work, and load testing under real concurrent traffic.
(Smaller items, EXIF auto-orientation, an embedded SPA dashboard, Firestore document windowing, are tracked in the README roadmap and not repeated here.)

Evaluation Methodology (how to read our numbers)

Metric / Dimension	Implementation Detail
Production cognitive scorer	Gemini (`gemini-3.5-flash`)
Evaluation harness	Deterministic (regex/assertions, zero LLM)
LLM-as-judge in eval suite	No
Memory A/B experiment	3-essay controlled scenario, n=1 student, not a statistical claim
Learning-outcome measurement	8 author-written thesis pairs, not 8 students; no control group; live-model scores, so re-runs move the numbers
Eval case falsifiability	Every case verified by sabotage test (ADR-019)
Persona fidelity, live mode	Opt-in `--live-persona` run, separate report: 2 of 4 personas drift into the Skeptic's register on a hard essay

We don't use an LLM to judge whether our own system passed its safety and behavioral tests. Gemini stays in its job as the production scorer; the eval harness checks deterministic output constraints and score deltas, nothing softer.