Provenance

Every student can submit perfect code now. Provenance shows you who actually learned

Comment

Inspiration Every instructor grading coding assignments right now is fighting a losing battle. A student can open ChatGPT, paste in the assignment, and submit a perfect solution in under five minutes — and it will look identical to a solution that took a genuinely engaged student 40 minutes of debugging, wrong turns, and learning to produce. Traditional plagiarism detectors like MOSS compare final code against final code, so they're blind to this: there's no duplicate to catch, because the "cheating" isn't copying, it's outsourcing the thinking.

I didn't want to build another tool that tries to ban AI use, because that fight is already lost and, frankly, misguided — AI-assisted coding is a real skill now. Instead I wanted to flip the question instructors ask from "is this code correct?" to "did this student actually do the work?" That's where the name came from: Provenance — not grading the artifact, grading where it came from.

What it does Provenance is a browser IDE for coding assignments with a built-in AI tutor, where every action is instrumented: every edit, every prompt sent to the AI, every AI response, every pasted block, every test run is logged as a timestamped event. When a student submits, an Assessor Agent replays the entire session timeline and produces a process narrative, an authorship score, an engagement score, and four viva questions generated specifically from that student's session — for example:

"In your early brute-force code, you checked len(set(sub)) == len(s). Why did that cause your function to return 0?"

Two students can submit byte-identical final code and score completely differently, because the platform grades how they got there, not just what they turned in.

How I built it The architecture is split into three intentionally decoupled pieces:

  1. Student Workspace — a React + Monaco IDE that fires a batched event stream to the backend (flushed every 5s or 20 events, whichever comes first), so the client never blocks on logging.

  2. Tutor Agent — a Socratic AI tutor that's deliberately never allowed to refuse — if a student demands the full solution, it gives it. The philosophy is "allowed but instrumented": accountability lives in the logging, not in restricting the AI. Every response gets auto-tagged server-side (CONCEPTUAL_HELP, DEBUG_HELP, PARTIAL_CODE, FULL_SOLUTION) so the Assessor can later see exactly how much of the solution the AI handed over.

  3. Assessor Agent — the core of the product, and a two-step pipeline. Step A is pure deterministic TypeScript with no LLM involved: it reduces the raw event log into a SessionSummary — meaningful-edit ratio (AST-token changes vs. whitespace), the fraction of the final code that arrived via a paste that fuzzy-matches a prior AI response, failure→fix cycles, struggle segments. Step B sends that summary plus the full transcript to an LLM with a strict JSON contract, validated with Zod and retried once with the validation error appended if parsing fails:

{ processNarrative: string, // 3-5 sentence story of how the student worked learningSignals: { signal: string, evidence: string }[], concernSignals: { signal: string, evidence: string }[], authorshipScore: number, // 0-100, how much of the reasoning was demonstrably the student's engagementScore: number, // 0-100, iteration, debugging, conceptual questioning vivaQuestions: { // exactly 4, each anchored to a specific session moment question: string, anchor: string, expectedUnderstanding: string }[] } One detail I'm proud of: the AI layer auto-detects whether the configured key is an OpenAI key (sk-...) or a Gemini key by shape, and transparently routes to the matching SDK — so I could develop the entire thing against a free Gemini key and swap to OpenAI with zero code changes.

For the demo, since a live judging session can't wait 40 minutes for a real "genuine" solve, I wrote a seed script that inserts two fully-formed, pre-recorded event timelines against the same assignment — one simulating a genuinely engaged 40-minute session, one simulating a 4-minute "solve this for me" outsourced session — and runs the real Assessor Agent against both at seed time, so the dashboard shows two real, LLM-generated reports with wildly different scores on identical final code.

Challenges I ran into Getting paste-detection to mean something. Flagging every paste event is noisy — students legitimately paste their own earlier code, error messages, etc. I had to build fuzzy substring matching between pasted content and recent AI responses so the signal is specifically "this text came from the AI," not "this text was pasted." Keeping the Assessor's output trustworthy. LLM JSON output isn't reliable by default, and a grading tool that hallucinates evidence is worse than useless. Wrapping the entire report contract in a Zod schema with a validation-error retry loop was non-negotiable. Making the viva questions actually session-specific. It was easy to get generic questions like "explain your algorithm." Getting the model to anchor each question to a literal moment in that session's diff history took real prompt iteration. Server-side code execution without a full sandbox. Running arbitrary student Python/C++ safely in the time I had meant leaning on a hard 5-second child-process timeout with piped stdin/stdout rather than building out real container isolation — a deliberate, explicit scope cut for a demo-grade build rather than something quietly skipped. What I learned The hardest part of this project wasn't the AI integration — it was realizing that a good process-assessment system is mostly deterministic data engineering (Step A) and only a thin LLM reasoning layer on top (Step B). The model is far more trustworthy and specific when it's handed a clean, pre-computed summary instead of being asked to reason over a raw event firehose. Do as much as possible in plain TypeScript, and only reach for the model once the hard part is already reduced to structured facts — that's the thing I'm carrying into the next AI project.

Built With

Share this project:

Updates