Sonny: The Socratic Sandbox

Students teach a confidently wrong AI. Sonny reveals what they truly understand, can apply, and remember

Comment

Inspiration

Growing up, our teachers often repeated the same core mantra: if you truly understand something, you should be able to explain it to someone else

Generative AI has made that test more important. Students can now produce polished answers without genuinely working through the reasoning behind them. The more thinking we “outsource,” the harder it becomes for teachers to determine what a student actually understands and where they are still struggling

We did not want to build another AI detector or proctoring tool. We wanted to turn AI use into an opportunity to think and make each student’s reasoning visible

That led us to Sonny. We flipped the usual AI tutor relationship on its head: the AI is not the expert teaching the student. Sonny acts like a classmate who does not understand the problem, and the student has to teach him. This turns a tutor-user interaction into a peer-learning exercise built around explanation and reasoning

The learner must explain the idea, defend the reasoning, apply it to a different case, and recall it later. Sonny evaluates those stages separately, then shows the student and teacher exactly which ideas were understood, contradicted, or still missing

Instead of using AI to avoid the work of learning, Sonny makes the reasoning itself the work.

What we built

A four-stage reasoning checkpoint

Each Sonny checkpoint examines understanding across four distinct stages:

  1. Explanation: Sonny presents a believable misconception, and the learner teaches him why it is wrong.
  2. Transfer Learning: Sonny changes the problem and asks the learner to apply the same idea in a new setting.
  3. Retrieval: The earlier conversation disappears, and the learner reconstructs the idea from memory.
  4. Inspection: The student and teacher see what was demonstrated, contradicted, or still missing, with every conclusion tied to something the learner submitted.

The Calculus I example

Our Calculus I demo begins with Sonny confidently claiming that (f'(c)=0) guarantees a maximum or minimum. The learner must teach Sonny why that claim is wrong, use (x^3) as a counterexample, explain what the sign of the derivative actually proves, and reconstruct the distinction later from memory.

Simply entering the correct rule is not enough. Sonny looks for the reasoning that makes the rule true and whether the learner can still apply it when the problem changes.

The student experience and growth recommendations

The checkpoint sits inside a broader learning platform with 4,500 practice prompts organized into 90 topics across six subjects. Students can explore a personalized knowledge map, identify strengths and growth areas, and receive evidence-based recommendations for what to practice next.

After each checkpoint, Sonny prioritizes contradictions, missing prerequisites, transfer gaps, and concepts due for retrieval. Its user-to-item model matches learner’s evidence with relevant problems, while its item-to-item model compares shared concepts, prerequisite distance, misconceptions, difficulty, and transfer value

Every recommendation includes the reason it was selected, its supporting evidence, uncertainty, and alternative problems. As of now, Sonny combines Bayesian Knowledge Tracing with spaced retrieval to guide practice without presenting its estimates as a diagnosis or definitive measure of mastery

The faculty experience

A separate faculty observability dashboard turns the same reasoning evidence into a clear view of class-wide misconceptions, individual learning histories, and possible next teaching actions. Instead of seeing only whether an answer was correct, instructors can see where a learner’s explanation held up, where it broke during transfer, and what should be revisited next class.

How it works

Sonny operates as a continuous learning loop: what happens during one problem determines what the student sees next

  1. Build the learner context: When a student logs in, Sonny rebuilds their learning state from previous evidence. The student sees a knowledge map, current growth areas, and a recommended next problem
  2. Run the checkpoint: The student chooses a problem and teaches Sonny through explanation, transfer, and retrieval. Each stage tests the same idea differently, so one correct statement cannot hide a later gap
  3. Interpret, then verify: On each turn, GPT receives the teacher-approved rubric, current stage, relevant conversation history, and learner’s latest response. It proposes which concepts were demonstrated or contradicted, identifies the supporting evidence, and drafts the next Socratic question. Before anything enters the learning record, deterministic code validates the result. It rejects unknown concepts and fabricated quotations, requires evidence from the learner’s current response, preserves later contradictions, and prevents invalid stage transitions A simplified evidence record looks like this:
{
  "concept": "critical_point_is_candidate_not_verdict",
  "explanation": {
    "status": "demonstrated",
    "evidence_turn": 3
  },
  "transfer": {
    "status": "contradicted",
    "evidence_turn": 5
  },
  "retrieval": {
    "status": "missing"
  }
}

In this case, the learner explained the concept correctly but contradicted it when the problem changed and has not yet reconstructed it from memory. Sonny preserves those differences instead of collapsing the session into one score.

  1. Update the learner model: Each validated observation updates student’s evidence history/log, concept graph, Bayesian Knowledge Tracing estimate, and spaced retrieval schedule. Lightweight learning process signals provide additional context, while conceptual credit remains grounded only in exactly what the student explained
  2. Close the loop: Sonny’s user-to-item (u2i) model matches the learner’s current gaps with relevant problems. Its item-to-item (i2i) model is responsible for comparing shared concepts, prerequisites, misconceptions, difficulty, and transfer value across hundreds of students. Every recommendation includes supporting evidence, uncertainty, and alternatives. The student sees an updated knowledge map and next step. Faculty see the same evidence organized into class-wide misconceptions, individual learning histories, and possible teaching actions ## How we used Codex and GPT-5.6 Sol TL;DR: By our estimate, Codex helped our two-person team build at least five times faster than our normal development pace. In one week, we moved from an idea to a deployed product ready for students and faculty

Development with Codex

We used /goal to break the project into clear milestones and keep long development sessions aligned. Codex helped us build the checkpoint state machine, evidence ledger, recommendation engine, student experience, faculty dashboard, deployment pipeline, and test suite.

The most valuable part was its ability to reason across the entire product. Sol could see how a change to the evaluator would affect retrieval, recommendations, teacher insights, and tests later in the system. This planning felt noticeably stronger than our previous GPT-5.5 workflows

Codex also challenged our implementation - it’s extremely conducive to work with an agent that is mindful of good engineering practices. It found cases where Sonny rewarded vocabulary without reasoning, reused evidence from the wrong stage, or missed a later contradiction. We turned each failure into a regression test instead of treating it as a one-time fix

Serving learners with GPT-5.6

Inside Sonny, GPT-5.6 uses the OpenAI Responses API to interpret a learner’s explanation, identify possible evidence, and draft the next Socratic question. Its role is intentionally bounded: deterministic code validates the evidence, controls checkpoint progression, and owns every saved conclusion.

The same architecture can run with a private GPT-OSS model on local compute. This allows schools to choose between frontier hosted intelligence and their own infrastructure without changing Sonny’s evidence standards.

The finished build passes 296 automated tests, including 16 reviewed learner journeys and 144 synthetic recommendation episodes.

What we learned

We have shared our prototype with several high school students in informal sessions. We saw students move from short answers toward explanations that used mechanisms, counterexamples, and self-correction. (Note: these weren’t controlled studies, so we are not claiming measured learning gains - but this served as a great feedback point for us to ground and improve the platform)

One thing stood out: students can have the right intuition before they know the formal vocabulary. Sonny has to recognize a good ordinary-language explanation without rewarding someone for blindly listing the right terms.

Honestly, the hardest part of the project was separating a helpful conversation from a defensible learning claim. A model can ask a great question without being trustworthy enough to control a student's record. We tackled this by ensuring Sonny’s model can listen and respond while the evidence system stays constrained and inspectable.

What's next

The next major feature is engaging with universities and enabling professors to upload a course objective, review the proposed misconception and rubric, approve the transfer and retrieval tasks, and publish the checkpoint for their students.

This would enable us to run a small voluntary classroom pilot with educator oversight. We would also like to integrate Sonny with tools schools already use, including learning management systems, Desmos, Khan Academy, and existing assessment workflows.

Our north star is simple: students are going to use AI. We want that use to help them become better thinkers instead of better answer collectors

Built With

Share this project:

Updates