VibeCheck

Inspiration

The way software gets written changed faster than the way it gets evaluated. Engineers ship with AI every day, and that's a good thing, but it means a finished piece of code stopped being proof of skill. Companies hiring engineers noticed it first: anyone can submit working code now; the real signal is who understands it. Universities have the same gap at much larger scale, with zero tools built for it. A submitted assignment no longer tells a CS professor who can program.

The two common reactions both fail. Banning AI is unenforceable, and it trains students for a job that no longer exists. AI detectors are worse: they produce false positives that punish honest students while missing real misuse. What actually works is making students explain their work. Universities across disciplines are reviving oral exams and process-based assessment for exactly this reason, and programming courses are where that approach matters most, since the code can't speak for itself anymore. It's also where it's hardest to pull off. Nobody has the staff to run an oral exam per student in a 300-person intro course.

So the question behind VibeCheck became: what if the oral exam could scale, and be grounded in exactly what each student did?

What it does

Students solve a coding task in a web IDE with a GPT-5.6 tutor in the sidebar. AI is allowed, encouraged, and captured: every edit, prompt, response, insertion and paste goes into an append-only event log.

From that log, the professor gets four things no submission can show:

1. A line-by-line attribution heatmap (human/AI-generated /human modified Ai-code(AI-modified), plus an AI-dependency percentage). This part is fully deterministic. The tutor is ours, every generation passes through our server, so we know what the model produced. When it lands in the editor it gets tagged as AI right there; if the student later edits it by hand, it becomes AI-modified. There's no ML detection involved and no false positives. It's bookkeeping, not inference.
2. A session replay timeline: the process, not just the result.
3. An automated oral defense. GPT-5.6 reads the student's final code, attribution ranges, prompts and event history, generates 3–5 questions about their specific decisions, then grades the written answers with per-question scores and reasoning. It also runs a debug challenge (find a bug injected into your own submission) and evaluates prompt quality.
4. A course dashboard with AI dependency and comprehension score for every student, side by side.

One thing I decided early: this is not a cheat detector, and I don't want it to become one. It assumes AI is a legitimate part of the work, like in industry, and grades what actually matters now: judgment, verification, comprehension. VibeCheck never auto-flags anyone. It hands the professor evidence plus a formative summary, and the professor decides.

How we built it

I spent the entire first hour not writing features but writing AGENTS.md: the event-log schema, the deterministic attribution rules, every API type, the security rules. That felt slow at the time. It turned out to be the whole trick. With the contracts frozen, I could run Codex agents two at a time on git worktrees (editor + event logging alongside the attribution engine + heatmap, then oral defense alongside the professor views) and they produced compatible code without stepping on each other. Every task prompt had the same shape: read AGENTS.md, here are the exact files you may touch, here are the acceptance criteria, here's what NOT to do. I reviewed each block before merging.

GPT-5.6 runs ~5 roles, all with zod structured outputs: the student's tutor, the oral examiner (question generation and grading as separate roles), a process analyst that narrates the session for the professor, the prompt evaluator, and the debug challenge. Stack: Next.js + TypeScript + Monaco, Supabase Postgres (server-side only, RLS deny-all), DB-backed rate limits that fail closed, token caps and retry-with-backoff on every model call.

Challenges we ran into

• Making attribution truly deterministic Edge cases everywhere: pastes matched verbatim against stored generations, insertions shifting line origins, edits inside AI blocks. I chose to document it as a known limitation instead of patching it, because the fix would have broken the mirror between the event log and the editor.

Accomplishments that we're proud of

The evaluator passed its hardest test. While writing answers for the five synthetic student profiles, I planted two subtle direction errors inside otherwise fluent responses: one about a <= vs < window boundary, one about a backwards clock jump. Honestly, I half expected the examiner to wave them through, because both answers sounded right. It caught both, scored each 2/3, and named the exact imprecision in its reasoning. Across the five profiles it produced the full 0–3 range, from solid mechanistic explanations down to evasion, and confident-sounding answers that got the mechanism wrong scored lower than plain answers that got it right. That's the whole bet of the product, and it held.

What we learned

The best hours of the sprint were spent writing AGENTS.md, not code. A frozen contract plus parallel agents outproduced anything I could have done sequentially, and it wasn't close.

Owning the AI assistant changes everything downstream. The moment every generation passes through your own server, attribution stops being a detection problem and becomes an accounting problem, which is a much better problem to have.

What's next for VibeCheck

Everything below is a new reading over data the event log already captures, not new architecture:

• Run-as-Signal: a sandboxed Run button that measures verification behavior rather than correctness. Did the student run their code before submitting? When it failed, did they read the traceback or just paste it to the AI?
• Task catalog (the schema is already multi-task) and a weighted comprehension composite each professor can tune.
• Adversarial probing (questions about flaws that aren't there), spec-compliance checking, and multi-agent rubric review, with several agents grading in parallel against the professor's rubric.
• Voice defense via the Realtime API.

CS education is shifting from writing code to directing, debugging and verifying AI output. Assessment has to catch up and measure those skills. That's the infrastructure we're building.

Built With

• codex
• gpt-5.6
• monaco-editor
• next.js
• node.js
• openai
• react
• supabase
• tailwindcss
• typescript