-
-
Self-calibrating evaluation: code checks vs LLM judge with calibration override
Inspiration
What it does
How we built it
Challenges we ran into
Accomplishments that we're proud of
What we learned
Inspiration
LLM-as-judge is everywhere now — agents evaluate their own output and call it "self-reflection." But the judge is the same model that wrote the answer. If LLMs hallucinate facts, they hallucinate quality scores too. Arize's own docs acknowledge this: "LLM judges can exhibit biases and unreliable behavior." I wanted to build the agent that actually does something about it.
What it does
Cruicible takes a research question, breaks it into steps, searches the web, and drafts sourced sections. After each section, it runs a mixed evaluation — deterministic code checks (real URLs? specific content?) alongside an LLM judge. If a section fails, the agent replans and retries.
The key feature: when the LLM judge claims high faithfulness but code evaluators disagree, Cruicible fires a calibration override — catching its own judge inflating scores. This is meta-verification: verifying the verifier.
Finished briefs require human approval before publishing, and approved research is stored in a persistent wiki that can be re-verified against fresh sources.
How I built it
- Google ADK for the agent framework with Gemini 2.5 Flash
- Arize Phoenix for full observability — every eval score, calibration event, and replan decision is traced as span annotations
- FastAPI backend with SSE streaming to a custom web UI (Tailwind + Alpine.js)
- The self-calibrating judge runs code evaluators in parallel with the LLM judge and compares their outputs to detect disagreement
Challenges I ran into
Getting the agent to reliably follow the full Plan→Search→Draft→Evaluate→Save sequence was harder than expected. Gemini's automatic function calling sometimes short-circuited the loop, skipping evaluation entirely. I had to make the agent instructions extremely explicit to prevent this.
The calibration trigger also needed tuning — too sensitive and it fires on normal score variation, too loose and it misses real judge hallucinations.
What I learned
LLM judges are unreliable in predictable ways. They consistently over-rate faithfulness when the text sounds well-sourced, even when the actual citations are weak or missing. Code-based checks catch this reliably. The combination of both is much stronger than either alone.
What's next
Re-verification on a schedule — facts rot over time, and old briefs should be automatically re-checked. The architecture supports it, I just need to wire up Cloud Scheduler.
Log in or sign up for Devpost to join the conversation.