Feedback Auditor

Catch feedback drift before students do—without automating a single grade.

Comment

Inspiration

I built Feedback Auditor because serious assessment workflows may use anonymous student work and more than one marking pass, but teachers still need a final consistency check before results are released. I wanted that last check to be inexpensive. Instead of asking an AI to re-grade every paper or repeatedly compare every pair of submissions, the tool extracts each submission once and cross-checks the structured evidence with deterministic rules. That reduces repeated model calls and token use while keeping the final decision with the teacher.

What it does

A teacher provides an assignment, rubric, anonymous student work, rubric scores, and the feedback already written. GPT-5.6 produces constrained evidence signals tied to verbatim excerpts. The server then rejects any signal whose student, rubric dimension, or quoted excerpt cannot be verified against the source text.

Four visible, versioned rules turn accepted evidence into review questions:

  • R1: similar evidence received meaningfully different scores;
  • R2: a similar issue was discussed in one feedback record but omitted from another;
  • R3: a score and the written feedback may describe different performance levels;
  • R4: a scored rubric dimension is not addressed in the feedback.

The teacher can inspect the highlighted evidence, add a note, confirm, resolve, or dismiss a signal, filter the review queue, and export a Markdown decision log. Feedback Auditor never changes a grade automatically, declares which teacher is correct, infers protected characteristics, or labels someone as unfair.

How I built it

The application uses Next.js, React, TypeScript, Zod, the OpenAI Responses API, and Structured Outputs. I deliberately separated the pipeline into three parts: GPT-5.6 structures evidence, TypeScript rules detect review patterns, and a human records the decision.

The public GitHub Pages site is static and exposes no API key or billable endpoint. It has two judge paths. The narrated-video fixture reproduces a fixed six-submission workflow with seven findings and 88% feedback coverage. A separate precomputed Codex audit contains six full fictional essays, 32 source-verified evidence signals, 13 deterministic review questions, and 63% feedback coverage. It is clearly labeled as precomputed and makes zero OpenAI Platform API calls from the public site. A self-hosted server deployment can run the implemented live GPT-5.6 extraction route with the owner's own key.

Codex helped me turn the original product idea into the design document, production implementation, tests, fictional evaluation data, deployment workflow, and narrated demo. GPT-5.6 matters at the evidence-extraction boundary; the consistency rules and teacher decisions remain transparent and deterministic.

Challenges

The hardest design problem was avoiding an automated-grader experience. A consistency signal can identify something worth checking, but it cannot know the classroom context or determine the correct grade. I rewrote the product language and workflow around neutral review questions rather than AI verdicts.

Evidence grounding was another challenge. Structured output alone is not enough, so every quoted span is checked against the exact fictional submission before it can reach the rule engine. I also needed a public demo that judges could test without receiving or exposing my API key. Keeping the original video workflow reproducible while adding the newer Codex analysis led to two explicitly separate demo paths.

Accomplishments that I am proud of

  • A complete teacher-controlled review workflow with zero automated grade changes.
  • A production evidence gate that accepted all 32 saved sample signals and rejected ungrounded or mismatched records.
  • Deterministic R1–R4 tests covering 13 review questions and all 24 submission-rubric coverage records.
  • A public, no-login demo that needs no visitor API key and does not upload student data.
  • A 2:44 English narrated demonstration with synchronized UI highlights.

What I learned

The most important lesson was that model-assisted education tools become easier to trust when model extraction, deterministic policy, and human judgment are visibly separated. I also learned that a precomputed public demo can still be honest and useful when its provenance and limitations are stated directly.

What's next

Next I would add CSV and LMS imports, configurable rule thresholds for different rubrics, multi-grader trend views, and institution-controlled server deployment. I would also run usability sessions with teachers to learn which signals save time and which need clearer explanations.

Live demo: https://breadtitor.github.io/feedback-auditor/
Source code: https://github.com/breadtitor/feedback-auditor

Built With

Share this project:

Updates