Eleza

A transparent oral-defense engine: it examines essays, code, lab reports, and case analyses in a live voice viva, shows why it asks every question, and hands teachers evidence, never a verdict.

Comment

Inspiration

AI made student work unfalsifiable. Teachers can no longer tell whether a student understands the work they submitted. Detectors misfire, worst of all for non-native English speakers, and the one assessment that still works, the oral exam, doesn't scale. Institutions are running vivas anyway, by hand, one student at a time, at a cost of dozens of instructor-hours per class.

What pushed this from observation to build was a second problem hiding inside the first: the AI oral-assessment tools emerging to fill the gap evaluate behind closed doors and return grades or flags. They replace an opaque student artifact with an opaque machine judgment. A tool that adjudicates academic integrity has to be more auditable than the work it examines, and none of them are. Eleza started as an answer to that specific failure: scale the conversation, and make the examiner show its work.

What it does

Eleza is a transparent oral-defense engine for submitted work. Give it an essay, a program, a lab report, or a case analysis, and:

  1. GPT-5.6 parses the submission into a graph of its own claims (design decisions, hypotheses, assumptions), every node anchored to exact character offsets. Questions target this structure, not the topic.
  2. A live viva follows, by voice or by typed answers through the identical path. The Realtime model speaks, but it never chooses a question. A separate GPT-5.6 examiner decides every routing move (probe, branch, advance) and its rationale streams into the margin as it decides. Every rationale must pass a schema-level gate, citing the target node and quoting at least five words of the student's actual answer, or be rejected and retried. Students control their own pace: an explicit Finish answer commit replaces pause detection, so pausing to think is never penalized.
  3. The output is a dossier, never a verdict. Exactly three finding types (cannot_reconstruct, mechanism_gap, inconsistency), each linking a transcript timestamp to the exact document span it concerns, with suggested questions for the teacher's own follow-up conversation. No authenticity score exists anywhere in the product. Conclusions belong to the teacher.

The engine is domain-parameterized: profiles define node vocabulary, edge semantics, and probe framing, while the invariants, gates, and divergence types stay universal across all four domains. The hosted demo caps sessions at about three minutes as a public-cost control; the assessment length itself is a deployment setting, built for teacher-configurable vivas of five to eight minutes.

And the examiner submits to its own method. Click Question this decision under any rationale and it must defend that routing choice, grounded strictly in its own records, with a plain refusal when asked beyond them.

How we built it

Built solo in seven days around a full-time internship, with Codex as the implementing collaborator in one primary session. Every prompt and the consequential decisions after it are preserved in an append-only BUILD_LOG.md: 36 entries, including the failures and corrections.

The build ran in acceptance-tested phases, each gated before the next: claim graph engine, then the voice loop and pure examiner, then the routing pipeline between them, then divergence analysis and dossiers, then the public judge demo and production hardening, then domain expansion via a behavior-preserving profile refactor verified byte-for-byte against pre-refactor prompt hashes, and finally the code, lab, and case profiles.

GPT-5.6 runs all three systems, with deliberate tier routing. gpt-5.6-sol generates claim graphs (runs once; everything downstream inherits its granularity) and post-viva divergence analysis. gpt-5.6-terra runs the latency-bound live examiner, with Sol as the retry tier when a rationale fails the quality gate. The gate doubles as the model router. The examiner call keeps a byte-identical prompt-plus-graph cached prefix across turns, hash-pinned by tests, with only the latest answer as fresh input.

The human contribution was the boundaries: five architecture invariants (the voice model talks, the examiner decides; one append-only decision log; schema-level rationale receipts; no register comparison; no verdicts), the manuscript-style visual language, and the rejected features. Audio retention was declined because storing student voice recordings is surveillance-weight infrastructure the product's privacy posture forbids. Transcripts carry the evidence without the biometric residue.

Every claim in this story is verifiable in the repository: the build log, the pinned hashes, the tests, and each fixture's documented weak spot with its expected finding.

Challenges we ran into

The voice model kept trying to be helpful. Early Realtime probes showed it answering the questions it was supposed to ask, violating the "voice talks, examiner decides" invariant from inside the model itself. The fix was architectural, not prompted harder: a versioned exact-delivery contract, with routing authority held entirely by the examiner's structured output.

Pause detection punished thinking. Semantic voice-activity detection cut students off mid-thought, exactly wrong for an exam about careful reasoning. We disabled VAD entirely and replaced it with an explicit Finish answer commit, so a student can pause as long as they need without triggering the examiner. Real use found this; no test did.

Our own tests caught our documentation drifting. A docs-only pass added metadata header comments to the versioned prompt files, which are rendered verbatim into the model payload, so "documentation" changed runtime bytes. Two hash-pinned snapshot tests went red on a fresh clone, contradicting a README claim of byte-equivalence. The resolution was diff-first: a verbatim comparison classified every changed line as non-semantic before the pins were updated, and the correction is logged append-only. The most serious bug in this submission was its test suite refusing to let a claim drift from reality.

The rules refused twice. Under the working agreement that invariants supersede mockups, Codex declined to reproduce landing-page marginalia whose wording implied verification the product doesn't perform, and separately declined to deploy a verified commit into a production environment missing its migrations. Both stops are logged as outcomes of the rules, and both were correct.

Accomplishments that we're proud of

  • Judges can reproduce our acceptance tests themselves. Each of the four fixtures ships with a documented weak spot and its expected finding: a program's unjustified data-structure choice, a lab report's overreaching conclusion, a case memo's unstated assumption, an essay's unreconstructable mechanism. Defend one poorly and watch the dossier point at the exact span.
  • The receipts explain themselves. Rationale specificity is enforced at the schema level, so a generic rationale cannot reach the screen, and the meta-viva means every judgment in the product can be cross-examined and refuses to confabulate beyond its records.
  • Bias designed out, not disclaimed. Written-versus-spoken register comparison is excluded by construction, since content reconstruction is register-independent, and the typed accessibility mode runs through the identical examiner path.

What we learned

Constraints on model behavior have to live in architecture, not prompts. The Realtime model's helpfulness reflex proved it in the first live probe. Interaction truths only surface under real use: no deterministic test would have found that pause detection punishes thinking. And an append-only log of prompts, decisions, failures, and corrections turned out to be more than documentation. It's what made the honest version of this story writable at all.

What's next for Eleza

Paste-your-own for the code, lab, and case domains (currently curated fixtures); longer configurable vivas in the hosted product; and pilot conversations with academic integrity offices, where a dossier of span-linked receipts is what a hearing panel actually needs. The larger thesis Eleza demonstrates, auditable AI judgment where the system produces evidence a human adjudicates, extends anywhere a document used to prove understanding and no longer can.

Built With

Share this project:

Updates

Submission history