i-learn

AI essay grading with dual-examiner precision, long-term student memory, and Socratic math tutoring — serving real IELTS and Gaokao test-takers.

Comment

Register with invitation code: 93XLLJ

Inspiration

In China, English writing prep is high stakes and feedback is scarce. A student grinding for IELTS or the Gaokao writes dozens of essays and gets back, at best, a number and a few red marks. A real examiner does far more: they read the essay as a whole, place it in a band, mark individual sentences, explain why each error matters, write a model rewrite one band up, and remember which mistakes you keep making.

i-learn tries to be that examiner. It runs in production, serving real TestFlight users, and it's built on one belief: an AI tutor has to teach, not just score, and it has to remember you from one essay to the next.

What it does

The flagship line is IELTS writing coaching, live in production. A student submits an essay as text or as photos of handwriting, and a background job runs a multi-turn grading session.

The coach agent reads its long-term memory of this student, places the essay against the official band descriptors, edits it sentence by sentence with diffs, writes a model rewrite one band up, extracts reusable expressions, and files a structured report. A second examiner agent, with no memory and no teacher persona, independently scores the same essay. The final score takes the higher of the two on each dimension. That policy came out of 11 rounds of calibration, which showed the model's bias is strictly one-sided: it only under-scores.

After grading, the student can ask follow-up questions (async jobs with live streaming text) and watch their score curve grow. The teacher's memory of them carries into the next essay.

The same agentic backbone powers two more lines. Gaokao English essay grading is the newest, built during Coding Week: the identical pipeline adapted to China's holistic five-band rubric, where you pick the band first and then place the essay within it, running a non-Claude model through an OpenAI-compatible function-calling transport. Gaokao math Socratic tutoring is also in production, with three teaching modes (fast, recognize, and invent-before-tell), a dual-axis student mastery model, and a spaced-repetition review queue that resurfaces due items through the homework-help flow.

The rest of the stack is the unglamorous part: invite-gated auth, per-student monthly AI budgets with a provable soft cap (if remaining budget is above zero, one more call is guaranteed to go through), self-built OTA updates, blue-green deployment, and an admin dashboard where every prompt, message, tool execution, and token count can be inspected.

How I built it

The development workflow itself is agentic. Claude Code orchestrates, and OpenAI Codex runs as the implementation subagent, doing a large share of the actual coding in parallel git worktrees. Two independent models cross-review every plan and implementation before merge, and a milestone ships only when both sign off. Under this workflow, the newest grading line went from an empty directory to a calibrated pipeline in days.

Each essay spawns a background job. The coach agent holds exactly two tools: memory, a DB-backed virtual filesystem with an index file plus one fact per file, where writes are staged in an overlay and committed only on success, and submit_report, which enforces a strict structured schema. The examiner session is deliberately stripped down: no teacher persona, no memory, a fully static prompt, pure scoring. Both scores are kept for observability.

The same session interface (run_session / make_tool) runs on claude-agent-sdk and on a hand-rolled OpenAI-compatible transport: tool-use blocks map to function calls, images become data-URI image_url entries, and SDK-managed turns become a manual loop. When the provider swapped, neither the agent code nor the eval harness changed a line.

Evals ran through the whole build. A fast scorer-only harness costs about $0.10 per case, and full-pipeline acceptance runs cover JSONL datasets spanning low, mid, and high bands. The IELTS grading doctrine went through 11 measured iterations, and every version's calibration results are documented.

On the ops side there are 114 backend test files covering migration idempotency, concurrency guards, and doctrine evals; OTA updates with two guardrails (a runtime-version fingerprint plus an env consistency check); and a db-guard wrapper that hard-blocks destructive SQL against production.

Challenges I ran into

The model systematically under-scores. Across 11 calibration iterations it rated strong essays 1 to 1.5 bands too low. I tried five different prompt strategies (anchor placement, top-down cumulative testing, mechanical error counting, dual-anchor ladders, and bias feedback) and every one hit the same ceiling, so I stopped treating it as a prompting problem. The fix was architectural: separate the scorer from the teacher entirely, then take the max of the two scores per dimension, which works precisely because the bias only points one way.

There are no gold labels for Gaokao essays. Unlike IELTS, publicly available band-scored Gaokao essays simply don't exist, so every "official score" in my dataset was one teacher's estimate. The first calibration run exposed this: a reference essay I had pegged at 20 came back as 16 or 17, reproducibly, within one point across three runs. The model's judgment was more consistent than my single-point labels. The bottleneck was labeler error, not model error.

Reasoning models don't respect tool discipline. The non-Claude model emits thinking turns with empty content, then tends to dump its entire analysis as prose without ever calling the submission tool, so the run "succeeds" with no report. Fixing it took both a doctrine rule (all grading output goes through the tool, never prose) and a transport-level nudge: when the runner sees text without a tool call, it injects a redirect message, up to three times.

And one silent schema collision: two product lines both defined a model named EssayOut, so FastAPI quietly prefixed the OpenAPI schema names and broke the frontend's typed references. A one-line namespace-prefix rename fixed it.

Accomplishments that I'm proud of

  • Live in production: real TestFlight users on the flagship line, a full eval harness, and 114 backend test files.
  • A measured answer to LLM scoring bias, backed by 11 documented calibration rounds rather than a hunch.
  • Vendor portability: the same pipeline runs on two different model providers with zero changes to agent or eval code.
  • Clean cross-band discrimination on the new line: high-band essays score 21 to 22, mid essays 16 to 17, low essays 8, and the results hold across runs.
  • A codebase largely written by agents, with Claude Code orchestrating Codex and two independent models reviewing every merge.

What I learned

  • LLM scoring bias is structural, not a prompting problem. When five different prompt designs hit the same ceiling, the architecture has to change.
  • Label quality matters more than I expected. Once the model is more consistent than the human labeler, further prompt iteration accomplishes nothing; the fix is better data, double-scored by multiple examiners.
  • IELTS and Gaokao scoring are less similar than they look. IELTS averages four dimensions in half-band steps, while Gaokao picks a band first and places the essay within it. They need different anchors and they fail in different ways.
  • Agentic development genuinely scales a solo builder. An orchestrator, an implementation subagent, and cross-model review shipped a new product line in days without dropping test coverage.

What's next for i-learn

  • A real gold-standard set: around 30 exam essays, double-scored by multiple examiners across all five bands, replacing the single-teacher estimates.
  • A dedicated test suite and admin observability UI for the newest line.
  • Structured input for continuation writing, with the source text and paragraph starters as separate fields, for tighter coherence checking.
  • Transferring the calibration method (dual examiner, max policy, eval harness) to other standardized writing exams.

Built With

Share this project:

Updates