DiffStory

The post-prompt receipt for AI coding agents that catches unrequested collateral code changes.

Comment

💡 Inspiration

Every developer using modern AI coding agents (Claude Code, Cursor, GitHub Copilot, Codex, OpenCode) has felt the sting of silent collateral damage.

You give the agent a simple, harmless prompt:

"Change the submit button color to blue"

The agent complies and turns the button blue—but quietly wipes out an essential email validation check 15 lines above. Because autonomous agents generate code so quickly, developers face a brutal dilemma: either blindly commit broken code or spend 10 minutes squinting at unified git diff outputs line-by-line.

We built DiffStory to act as a post-prompt receipt for coding agents: a verifiable accounting system that checks what you asked for against what the agent actually changed.

⚡ What It Does

DiffStory is a single-screen developer workbench that audits unified diffs against user prompts using a hybrid architecture:

  1. 100% Deterministic Token Matcher: A pure JavaScript token intersection engine splits the unified diff on @@ -X,Y +A,B @@ headers and extracts mutated lines (+ and -). It splits camelCase, filters syntax boilerplate, and compares the changed lines against your prompt.
  2. Zero-Hallucination Classification: Every hunk is deterministically categorized as either:
    • MENTIONED IN YOUR PROMPT (e.g., your blue button change).
    • NOT MENTIONED IN YOUR PROMPT (e.g., deleted email validation).
  3. Literal Word-Level Receipt: Gives you cryptographic-like proof: > "Your prompt mentions 'submit', 'button', 'color', 'blue'. Nothing in your prompt mentions 'email', 'includes', 'seterror'."
  4. Deterministic Priority Sorting: Automatically sorts amber warning hunks to the top of the feed so collateral damage is immediately visible without scrolling.
  5. Bounded Live AI Remediation: Google Gemini (gemini-3.5-flash-lite) is queried strictly on unmentioned hunks to generate:
    • A 1-sentence plain-English impact statement explaining what broke.
    • A 1-click copyable Restore Prompt to paste right back into your coding agent to undo the damage.

🛠️ How We Built It

  • Architecture: Built following the Devpost Learn Skill Pack specification (scope.md, prd.md, spec.md, and app-map.html).
  • Frontend: React 18, Vite, Tailwind CSS, Lucide Icons, designed with a tactile archival paper aesthetic (#FAF8F5) avoiding generic purple AI gradients.
  • Deterministic Core: Pure JavaScript tokenization engine with camelCase splitting, syntax stripping, and set intersection.
  • Bounded Generative AI: Google Gemini API (gemini-3.5-flash-lite / gemini-3.8-flash) returning strict JSON with graceful offline fallback.

🧗 Challenges We Ran Into

  • The LLM Hallucination Trap: We initially considered asking an LLM to classify whether a diff matched the prompt. We quickly realized LLMs hallucinate justifications (e.g., "The user asked for blue, so removing email validation makes the form simpler"). We solved this by strictly delegating the classification gate to deterministic JavaScript code, reserving the LLM solely for impact summarization.
  • Code Token Isolation: Raw diffs contain language boilerplate (const, return, function, className) that would cause false matches. We implemented custom AST-level keyword stripping to ensure only domain tokens (email, submit, button, blue) are matched.

🏆 Accomplishments We're Proud Of

  • 0% Classification Hallucinations: Verifiable classification proof that judges and developers can inspect in under 5 seconds.
  • Sub-Second Performance: Client-side parsing that classifies hundreds of diff lines instantaneously.
  • 1-Click Remediation: Moving beyond passive diff viewers to active healing with ready-to-paste restore prompts.

📚 What We Learned

The Deterministic-First AI Pattern: AI agents shouldn't be asked to act as judges of factual truth when deterministic algorithms exist. By building deterministic verification gates first and bounding the LLM to semantic synthesis, software becomes predictable, auditable, and production-ready.

🔮 What's Next for DiffStory

  • CLI Pipe Tool: Running git diff | diffstory directly in the terminal before committing.
  • VS Code / Cursor Extension: In-editor post-prompt receipts that appear immediately after an agent finishes generating code.

Built With

  • ai-agents
  • code-review
  • developer-tools
  • gemini-api
  • git
  • git-diff
  • google-gemini
  • javascript
  • lucide-react
  • node.js
  • open-source
  • react
  • tailwindcss
  • vite
Share this project:

Updates

Submission history