LLMVibeCheck

Every run is scored by hidden tests in a sandbox, plus a judge panel from three companies. No model grades only its own family.
Your dashboard: every squad, task and run in one place, with total spend tracked to the cent.
Six squads solved the same 24-test task. Solo Claude Sonnet won 95/100 for $0.019 in 11s. More models didn't mean better results.
Build a squad: Claude plans, GPT builds, Gemini reviews, and the builder loops until the reviewer approves.
Watch a squad work live. Every step's prompt, answer, tokens, cost and time are recorded as it happens.

Inspiration

Everyone has a favourite AI model, but real work is increasingly done by teams of models: one plans, one writes, one reviews. We kept asking: which combination is actually best for my job? Is a mix of Claude, GPT and Gemini better than one strong model alone? Is a cheap model plus a strict reviewer good enough? Nobody could tell us with evidence instead of vibes. So we built a way to check the vibes.

What it does

LLMVibeCheck lets you build "squads" of LLMs from different companies, run them on the same task, and see which squad gets it right for the least money and time.

Build: drag models into roles (planner, builder, reviewer, or your own) on a visual canvas, with loops ("send it back until the reviewer approves"), human checkpoints, reusable skills, and voice steps.
Run: watch every step live. Each prompt, answer, token count, cost and latency is recorded.
Evaluate: each run is scored three ways:
- Hidden tests run in a locked-down sandbox
- A judge panel of three models from three companies (Claude, GPT, Gemini), so no single model grades its own family
- Optional human ratings
Compare: a leaderboard ranks squads by overall score, quality, cost, speed, or "cheapest correct answer," with a quality-vs-cost chart.

How we built it

Frontend: Next.js + React + Tailwind, with React Flow for the squad canvas. Deployed on Vercel.
Backend: FastAPI with asyncio and server-sent events for live progress. Deployed on Render with Docker.
Data: TimescaleDB (TigerData), so every message and step is a time-stamped record we can aggregate.
Models: OpenRouter for one API across companies, with live per-call cost. ElevenLabs for voice steps.

Scoring is on a 0–100 scale. Quality is a weighted average of whatever signals exist:

$$Q = \frac{w_j \cdot 10J + w_t \cdot 100T + w_h \cdot H}{w_j + w_t + w_h}$$

where $J$ is the judge panel score (0–10), $T$ is the test pass rate, and $H$ is the human rating mapped to 0–100. Efficiency is relative to the best squad on the same task (cost score $= 100 \cdot c_{best}/c$, and the same for speed). The overall score is

$$\text{Overall} = Q \cdot \left(1 - \min(1,\ 0.2\,\Delta_{cost} + 0.1\,\Delta_{speed} + 0.1\,\Delta_{effort})\right)$$

so a cheap-but-wrong squad can never beat a correct one.

Challenges we ran into

The reviewer stole the answer. The final output of a squad was the reviewer's "APPROVED" message, not the code. We now trace back to the actual deliverable.
Reviewers that never say yes. In one mixed squad, a strict reviewer looped the builder over and over. Same correct answer, but 2.8× the cost of a similar squad. Great finding, painful bill.
Our score went negative. Our first formula subtracted cost and time penalties, so a cheap failure could score below zero. Or worse, above a correct answer. We redesigned it as quality × (1 − penalty).
Keeping the tests honest. Generated code could print a fake "all tests passed" line. We added a secret per-run marker, JUnit XML parsing, and a sandbox that never sees API keys.
One judge isn't fair. A single judge model tended to favour its own style, so we built a cross-company panel and report how much the judges disagree.
Hackathon Wi-Fi blocked our database port, so we migrated the data over a phone hotspot at 5 a.m.

Accomplishments that we're proud of

A real experiment on a 24-test expression evaluator: 3 cross-company squads vs 3 solo models. All six passed 24/24.
- The most expensive squad cost over 6× more for the same result.
Finding that more models ≠ better results on well-defined tasks. That's exactly the kind of insight the tool exists to surface.
116 unit tests, which caught and fixed 5 real bugs before the demo.
A fully hosted app with a spending cap, plus a plain-language Docs page for non-technical users.

What we learned

Multi-agent setups add cost and latency fast. Their value depends heavily on the task and on how strict the reviewer is.
Evaluation is the hard part. Tests catch correctness, judges catch quality, and neither is enough alone.
Normalising scores relative to the best competitor makes results far easier to explain than raw dollars and milliseconds.

What's next for LLMVibeCheck

Repeat runs with confidence intervals, so rankings reflect consistency, not luck.
A benchmark library of tasks (coding, writing, data extraction, agents with tools).
Auto-search: given a task and a budget, automatically find the best squad.
Shareable leaderboards so teams can publish "which AI team works best for X."