DidacticOS

DidacticOS: an AI tutor that continuously learns how to teach and improves pedagogy without becoming a black box.

Comment

DidacticOS: a tutoring system that improves its teaching knowledge

DidacticOS is a Pedagogical Operating System for AI tutoring or for Humans to learn best strategy. Its central bet is deliberately different from “fine-tune the model after every interaction.” The language model remains replaceable. The durable asset is a governed body of teaching knowledge: what misconceptions matter, which strategies help which learner situations, what evidence supports that claim, and whether the evidence is strong enough to promote the strategy.

The system therefore separates four jobs:

  • The Runtime Kernel teaches in a live turn.
  • The Teacher Learning Kernel studies completed evidence offline.
  • The Pedagogical Knowledge OS stores versioned teaching knowledge.
  • A Foundation Model generates language through a provider adapter.

This is a hackathon prototype, but the architectural boundary is serious: fast tutoring stays predictable, while slower learning and validation occur outside the student-facing request path.

How Codex built it: a long-running governed loop

DidacticOS was coded in Codex while the brainstorming and architecture was done in collaboration with chatgpt during the hackathon, using the full 2,500-credit allocation. The project was not treated as one prompt followed by a large unreviewed patch or vibe-coded. We engineered a persistent control loop around Codex: a task ledger chose what could happen next, specialist agents handled distinct roles, explicit acceptance decided whether work counted, and a human could see and interrupt the loop at any time.

SELECT_TASK → PLAN → IMPLEMENT → VERIFY → ANALYZE → REPLAN
                                           ↘ DONE or BLOCKED

IMPLEMENTATION.md supplied ordered dependencies, planning requirements, acceptance criteria, and phase Definition-of-Done gates. AGENTS.md supplied the architectural invariants. For non-trivial work, Codex routed planning to a GPT-5.6 Terra specialist at xhigh reasoning effort; a separate GPT-5.6 Terra specialist implemented the approved plan at high effort; another high-effort specialist independently verified the named acceptance tests. Routine orchestration used medium effort. This adaptive routing put the most effort into later multi-component safety, validation, and recovery work rather than spending the same budget on every mechanical edit.

Circuit breakers, escalation, and acceptance

Circuit breaker was applied when a task was blocked after three failed verification attempts, after 30 minutes of ordinary active work (90 minutes for planning-heavy work), or earlier when two consecutive verification attempts produced the same failure signature. Every retry had to state a new hypothesis; otherwise it was escalation, not a fresh attempt. A blocked task, an ambiguous student-facing design choice, a one-shot benchmark claim, or a failed validation gate stopped autonomous progression and surfaced the issue for human review.

Specialist work used a receipt protocol: dispatch, completion, and exactly-once handoff consumption were append-only events. A task could not become done after a failed verifier, and the loop could not move on with an unconsumed specialist completion. Once a task passed and was committed, the loop selected the next eligible task automatically; it paused only at an explicit stop-and-ask condition, a blocker, an operator halt, or a phase checkpoint.

The monitoring UI

We also built a local monitoring UI for the loop itself. Its persistent left pane shows the phase/task roadmap and each task’s pending, doing, done, or blocked state; its central pane streams readable recent activity instead of raw JSON dictionaries. It exposes the current task, blockers and their reasons, halt/resume controls, and task-level token attribution. It tails the append-only .loop/events.jsonl stream, while a separate Langfuse panel handles model-call telemetry without mixing it into product evidence. The recorded build history contains 872 events across 68 task IDs: 145 specialist dispatches, 148 completions, 145 consumed handoffs, and 158 test runs.

Human Reviews

Autonomy did not replace human architectural authority. MathDial has no timestamps suitable for temporal filtering. An early automated proposal to filter records without timestamps would have removed the whole corpus; human review rejected it. The adopted design preserves the data, explicitly marks temporal evaluation unavailable, and reserves a timestamped cohort for future work. Naive chunking as suggested by codex was rejected by human overseer for more nuanced topic guided clustering. The same acceptance discipline preserved the consumed MathDial claim instead of silently rerunning or weakening it. This was the useful division of labor: Codex accelerated execution, while human review retained authority over irreversible data, benchmark, and safety decisions.

MathDial protocol, evidence boundary, and final verification

MathDial is the project’s verification contract, not merely sample data. Its pinned Hugging Face revision (acc3878459e0bd8c04ab840056572f0b8b1abe1f) is deterministically split by question ID, seed 20250717, into disjoint evidence roles: 429 records (15%) for fast loop-engineering checks, 2,003 (70%) for offline teacher-learning evidence, and 429 (15%) for held-out acceptance. The manifest records source hashes and selected IDs and refuses overwrite. This makes MathDial the common benchmark boundary for per-task verification, phase-level Definition-of-Done checks, and the original final-acceptance gate—while preventing development or learning work from leaking into the held-out slice.

Only the question and incorrect student attempt enter the Runtime. Ground truth, self-correctness, conversation, and annotations never enter the live decision. For the offline demo, a terminal self-correctness label becomes a documented Rasch proxy only after the Runtime turn (“No” is non-success; any other non-empty label is success). This lets MathDial exercise the whole evidence pipeline without pretending that a corpus label proves DidacticOS caused a learning outcome. MathDial has no usable timestamps, so temporal evaluation is explicitly unavailable; a future timestamped cohort can use the same protocol.

The benchmark discipline was tested for real. The one-shot MathDial held-out claim was consumed before a grouped-QID loader defect rejected 429 valid rows as 166 groups. Rather than mutate the split or weaken acceptance, the project preserved the incomplete claim. A separate pinned ten-group Eedi train cohort then completed the structural double-Runtime evidence recovery under the final acceptance protocol. That Eedi result is not a substitute MathDial outcome benchmark or a learning-effectiveness claim; it shows that the gate preserved its meaning under failure.

Architecture: stable Runtime, offline evolution

The live path is a microkernel-style supervisor with explicit workers; the offline Teacher Learning Kernel is a blackboard-style set of specialists that read durable evidence and write their own artifacts. They do not run inside a student turn or directly call one another. This separation keeps expensive reflection out of the request path and prevents an observation from becoming a production teaching rule without the candidate → validation → promotion path.

What happens in one tutoring turn

A tutoring turn follows one fixed, graph-directed path:

Student input
  → Central Orchestrator
  → Working Memory
  → Student Model
  → Policy Engine
  → Prompt Compiler
  → Foundation LLM
  → Output Guardrail
  → Student

The orchestrator is a microkernel-style supervisor; Student Model, Policy Engine, and Prompt Compiler are workers with explicit interfaces. LangGraph expresses this path as a state machine, so orchestration decisions can be deterministic even when the final response is generated by a model.

The Runtime Kernel does not run reflection, skill discovery, or model retraining. That matters for student safety and latency: the live system should not become more experimental because a learner happens to be using it.

Every turn produces append-only evidence rather than a silent black-box decision. In the current replay artifact, each real runtime turn has a Pedagogical Decision Record, telemetry record, counterfactual-policy record, session log, and guardrail decision. The guardrail is a hard graph node. If the response violates the teaching policy, the deterministic fallback is used rather than making another uncontrolled model call.

The Rasch Student Model in plain English

DidacticOS uses a deterministic Rasch-style IRT baseline: learner ability theta is compared with item difficulty beta to estimate a success probability.

P(correct) = logistic(theta − beta)

After an outcome, the state moves by a bounded residual update. It is an inspectable policy signal—not a claim of full psychometric calibration or proof that a generated response caused learning.

What “continuous teacher learning” actually means

Continuous learning here does not mean secretly changing the live tutor after every conversation. It means the Teacher Learning Kernel continually turns completed, immutable evidence into candidate pedagogical knowledge through an offline lifecycle.

The lifecycle is designed to answer a sequence of increasingly demanding questions:

  1. What happened? The system ingests completed Runtime telemetry and decision records. Evidence is append-only, so later analysis does not rewrite history.

  2. Did the learner appear to gain something? The Learning Gain Estimator compares the before and after learner-state estimates. In the current baseline, it deliberately uses learner-state evidence only and does not claim that a particular prompt caused the change.

  3. What teaching choice is plausibly responsible? The Attribution Engine looks for comparable alternatives using pre-treatment features: learning objective, learner profile, detected misconceptions, and prerequisite state. If there is no adequate comparison, it returns insufficient evidence rather than fabricating a causal ranking.

  4. What should we learn from the turn? The Reflection system records whether the selected strategy appears to have worked, failed, or missed an alternative. The Failure Analyzer classifies only evidence-backed categories, such as a guardrail failure or no model response; it does not invent latent explanations.

  5. Is there enough evidence to propose a reusable skill? Skill Discovery creates a Candidate Skill only when the effect is positive, the confidence interval clears zero, the evidence has no disqualifying failure, and the provenance is attached. A candidate is not production knowledge.

  6. Can the candidate survive validation? Candidates pass through offline simulation, benchmark, safety, statistical, and governance checks. Validation creates evidence about a proposed version; it is not allowed to edit a released skill in place.

  7. Can it be promoted safely? Only a promoted candidate becomes an immutable validated-skill version. The continuous-evolution control plane then publishes a new immutable snapshot after the Phase 5 promotion contract is satisfied. Snapshot publication is a controlled deployment event, not an incidental database write.

  8. Can we explain and reverse it? Lineage ties a skill version back to its evidence. Promotion, activation, deprecation, rollback, and retirement are lifecycle events with append-only receipts. If a later evaluation exposes a problem, rollback moves the active pointer to a previously valid immutable version; it does not erase history.

That is the core distinction in DidacticOS: the Foundation Model executes language, while the offline kernel improves the governed teaching policy around it. A released skill is immutable; improvement means proposing a new version with new evidence, never editing production pedagogy in place.

From data to a validated teaching rule

The full intended pathway is:

Immutable Runtime evidence
  → learner-state / gain analysis
  → comparable-cohort attribution
  → reflection and failure analysis
  → Candidate Skill with provenance
  → offline validation gates
  → immutable validated-skill version
  → promoted active snapshot
  → monitor, audit, rollback if needed

At each arrow, DidacticOS asks for a different form of evidence. A gain estimate is not causal attribution. Attribution is not a candidate. A candidate is not validated. Validation is not deployment. This may sound procedural, but it is how the project avoids allowing a single attractive interaction to mutate production pedagogy.

Demonstrable results, stated honestly

The included substantive local replay artifact contains 184 completed real Runtime turns. It contains 184 decision records, 184 telemetry records, 184 counterfactual-policy records, 184 session logs, and 184 guardrail logs. All 184 recorded guardrail decisions passed. With the documented terminal-outcome proxy, the mean corpus-derived Rasch change was +0.14272.

The artifact also shows why the validation gates matter. The policy recorded 368 rejected-strategy reasons (two rejected alternatives for each of 184 turns), and the run produced zero candidate skills and zero lifecycle receipts. That is expected from a small observational replay with strict evidence rules. A system that creates a polished new teaching rule from inadequate comparison data would be much less trustworthy.

Engineering choices that make the prototype reusable

The prototype has ports-and-adapters(Hexagonal architecture) boundaries around the Foundation Model. It can be demonstrated with a local Ollama model or an OpenAI-compatible API through the replay scripts, without importing a vendor SDK across the application. Presentation scripts place results in a results directory, while the larger learning-batch script can exercise a substantive MathDial cohort and offline analysis.

The system also distinguishes operational repair from semantic relaxation. For example, missing terminal labels are skipped with an explicit reason rather than treated as a successful or failed learning event. If the local provider raises a Foundation Model error during the learning batch, the script preserves completed artifacts, stops cleanly, and records the provider error in its summary rather than fabricating a result.

What is complete, and what remains honest work

A large portion of the implementation roadmap is complete: the runtime path, provider adapters, working memory, Rasch baseline, policy and prompt compilation, guardrail, evidence contracts, offline learning specialists, candidate and validation scaffolding, lifecycle controls, monitoring, replay tooling, and documented recovery artifacts.

The original MathDial final-acceptance claim remains permanently incomplete, while the separate Eedi train-cohort Runtime/evidence recovery is independently complete under its explicit exception. The P6-T9 learning-batch implementation was also independently verified and is marked done in the working ledger. The project was completed tightly against its implementation and structural-runtime goals, but it should not be presented as having proven pedagogical efficacy on a clean causal benchmark. It needs larger completed runs, repeated provider and regression testing, and stronger outcome evaluation—ideally on a suitable untouched or timestamped cohort—before making claims about improved learner outcomes. What it does demonstrate today is a functioning, safety-conscious architecture with end-to-end evidence flow and strict non-promotion when the evidence is insufficient.

Why this matters for a hackathon

Most AI tutor demos show a chat interface and a clever prompt. DidacticOS demonstrates the surrounding operating system that makes improvement governable:

  • a learner model that is explicit and inspectable;
  • a policy layer that chooses a teaching strategy;
  • hard safety guardrails;
  • immutable decision evidence;
  • an offline learning kernel that can propose, validate, version, promote, audit, and roll back teaching knowledge;
  • fixed corpus discipline;
  • and a Codex harness that treats planning, delegation, verification, acceptance, escalation, and observability as part of the engineering product.

The strongest claim is not “the model learned to teach perfectly.” It is that DidacticOS provides a disciplined architecture for continuously improving teaching behavior without allowing every model interaction to rewrite the tutor.

References

Built With

  • codex
Share this project:

Updates