VibeCheck

Check AI coding claims against a fixed GitHub commit, review requirements, and share the evidence.

Comment

Inspiration

An AI coding handoff often mixes statements that source can support with statements that need a real execution record. “The function exists” and “all tests passed” are different kinds of claims. Solo builders and reviewers need to see what supports each one before accepting “done.”

VibeCheck turns that handoff into an inspectable review, with evidence and a clear next step.

What it does

Paste an agent's message, enter a claim list, or import a GitHub PR or commit. VibeCheck inspects a fixed source revision, checks cited quotations, and shows each finding as Source supported, Partial support, Needs evidence, or Planning conflict.

Requirements can come from planning documents or explicit user input. The requirement view highlights conflicts and statements the handoff did not mention; an unmentioned requirement is a review prompt, not proof of missing implementation. Reviewers can copy a repair handoff, recheck a revision and compare findings with explicit context checks.

Reports include source locations, missing evidence and follow-up actions. They can be exported as JSON or Markdown and shared through a signed report link. Signature verification detects changes to the issued report; it does not certify the model's reasoning.

The demo uses three claims about Chalk at one pinned commit. Its MIT-license prohibition is a deliberately supplied review constraint, not Chalk's own policy. A source declaration can support the function claim, the manifest can conflict with the supplied constraint, and a tests-passed claim still needs execution evidence.

How it was built

The Devpost Learn Skill Pack guided the scope, PRD, technical specification, build and shipping workflow. The generated planning documents remain in the public repository's devpost folder.

The application uses Next.js, React, TypeScript and Tailwind CSS, with Google Gemini for claim extraction and assessment, GitHub REST API for source retrieval, and Vercel hosting. Server checks bind evidence to one commit and validate source quotations. Bounded provider attempts expose failures instead of substituting illustrative results. Ed25519 signatures cover the complete issued report, which can also be verified independently.

OpenAI Codex assisted implementation, debugging, review and verification. Standard frameworks, SDKs and licensed fonts are documented in the repository. The demo narration was generated with ElevenLabs.

Challenges

The hardest part was keeping the evidence boundary honest: a dependency declaration does not prove an integrated feature, a source file does not prove a deployment, and a test script does not prove a passing run. Other challenges included bounded source retrieval, exact quotations, malformed model responses, revision consistency and rejecting modified reports.

Accomplishments

The public application supports a complete handoff-to-report workflow. The release checks passed 37 production tests, 14 controlled browser workflows and 15 manufactured adversarial cases. These are engineering checks, not general model-accuracy measurements. Actual provider observations, including failures, are retained with their limits in the build and evaluation records.

What was learned

Planning before building made the product boundary explicit. Evidence that looks persuasive can still be too weak for the claim. Source support, observed execution and report integrity must stay distinct, and failures need to remain visible.

What's next

Independent first-use testing and broader blinded evaluation are the next priorities, followed by measuring whether the review helps real builders correct consequential handoff mistakes. Those outcomes are not yet established. VibeCheck remains a review aid: it executes no repository code, and human judgment is still needed.

Try VibeCheck · Source and setup · Planning documents · Evaluation and limits

Built With

  • devpost
  • github-rest-api
  • google-gemini
  • google-genai-sdk
  • learn
  • next.js
  • node.js
  • playwright
  • react
  • skill
  • tailwind-css
  • typescript
  • vercel
Share this project:

Updates

Submission history