Vibe Check

• 

Inspiration

I built Vibe Check which is an Agent Test Harness after noticing a gap in how AI agents are tested. Most tests check whether the final answer looks correct. But agents can produce convincing answers while skipping the retrieval, policy, or safety step that should have justified the answer.

I wanted a developer tool that treats agent behavior as a testable contract: not only “did the agent answer correctly?” but also “did it use the right tools?” and “did it gather the right evidence before answering?”

What it does

Vibe Check is a semantic testing framework for LangGraph agents.

Developers define tests in .agent-test.yaml files with a task, expected tool behavior, required evidence sources, an output rubric, and an optional reliability threshold.

Each run is scored independently on:

• Process — Did the agent call the expected tools in the expected order? Did it call a forbidden tool?
• Evidence — Did the agent gather grounding before answering? Did it use required sources?
• Output — Did the final answer satisfy a natural-language rubric?

It also supports repeated runs and minimum pass rates, so developers can test agent reliability rather than trusting a single successful run.

The results appear in the CLI and directly inside VS Code’s Test Explorer, with scorer-level results, visual summaries, and one-click traces.

How I built it

I built Vibe Check as a Python CLI and a VS Code extension.

The Python side loads YAML test suites, runs LangGraph agents with callback tracing, captures tool calls and outputs, and produces a JSON report. The process scorer uses ordered tool-trajectory matching, the evidence scorer validates grounding and required tools, and the output scorer evaluates responses against a natural-language rubric.

The VS Code extension consumes the JSON report and turns it into native tests with separate process, evidence, and output children. It also opens an automatic result summary after each run and lets developers jump directly from a failed result to the exact tool trace.

To demonstrate the product, I built a fictional Support Desk refund agent. Its skip_policy failure mode creates a realistic silent regression: the agent still issues a correct-looking refund, but skips the required policy lookup. Vibe Check catches this as:

process: fail
evidence: fail
output: pass

Challenges I ran into

The hardest challenge was making failures actionable instead of merely detectable. A red test is not enough if a developer still has to search logs to understand what happened.

I solved that by preserving the full tool timeline and connecting failed result cards directly to a trace view. Developers can immediately see which tool was missing, which tool failed, what evidence was returned, and what final answer the agent produced.

Another challenge was non-determinism. Agent behavior can vary between runs, so I added repeated execution and pass-rate thresholds:

$$ \text{pass rate} = \frac{\text{passing runs}}{\text{total runs}} $$

I also designed offline fallbacks and a clean bootstrap path so the project remains testable without requiring live API calls.

Accomplishments that I'm proud of

I am proud that the project goes beyond answer evaluation and makes process and evidence first-class testing signals.

I am also proud of the developer experience:

• Tests appear in the VS Code Test Explorer.
• Results are understandable without reading raw JSON.
• Failures explain what failed and why.
• Developers can open the exact tool trace in one click.
• The Support Desk demo shows a failure that answer-only testing would miss.

Most importantly, the project turns a subtle agent reliability problem into something concrete, visual, and fixable.

What I learned

I learned that agent quality cannot be represented by one score. Process, evidence, and output fail independently, and separating them gives developers much better debugging information.

I also learned that the best developer tools fit existing workflows. VS Code is where developers already run tests and investigate failures, so surfacing agent evaluations there makes the tool more useful than a separate dashboard.

Finally, I learned that repeatability matters. Agent tools need to acknowledge flakiness, not hide it.

What's next for Vibe Check

Next, I want to expand support beyond LangGraph, add richer evidence-grounding checks, and make test suites easier to generate from existing traces.

I also plan to add CI integrations, regression comparisons between agent versions, and stronger policy assertions for safety-critical workflows.

The long-term goal is to make semantic agent testing as routine as unit testing: fast, understandable, and part of every developer’s normal workflow.

Built With

• agent
• ai
• api
• developer
• evaluation
• groq
• json
• langchain
• langgraph
• llm
• openai
• pydantic
• pytest
• python
• pyyaml
• semantic
• streamlit
• tools
• typescript