ClinAgent Arena — Open Clinical AI Safety Leaderboard

Open-source adversarial red-teaming for clinical AI agents — mapped to EU AI Act risk, audit, and robustness requirements.

Comment

Inspiration

Security has had CVE and public vulnerability disclosure for decades — anyone can check if a piece of software has known weaknesses before deploying it. Clinical AI has nothing equivalent. Every hospital, startup, and regulator evaluating a clinical LLM or agent system is doing so blind, trusting vendor claims with no shared, independently-run benchmark to check against. With the EU AI Act now classifying most clinical decision-support systems as high-risk and requiring documented risk management, robustness testing, and audit trails, that gap felt urgent rather than theoretical. We wanted to build the tool we wished existed the last time we red-teamed a clinical AI system for a client and had to invent the methodology from scratch.

What it does

ClinAgent Arena adversarially red-teams clinical LLMs and agent systems against a shared, versioned attack taxonomy — unsafe dosage advice, hallucinated citations, prompt injection via patient notes, authority-framed jailbreaks, demographic bias drift, and poisoned RAG context — then scores every run through an independent judge pipeline. Results are published on a public leaderboard as a risk matrix, so a procurement team, developer, or regulator can compare systems on the same footing. The demo page lets anyone watch the attacker → target → judge pipeline run live against sample scenarios, no setup required, and the CLI lets any team drop the same eval suite into their own CI pipeline in minutes.

How we built it

We built the evaluation engine in Python on top of LangChain and LangGraph, with three cooperating agents — an attacker that generates adversarial cases from the taxonomy dataset, a pluggable target adapter that wraps whatever agent framework a team is using, and a judge that combines rubric scoring with NLI-based faithfulness checks from Hugging Face Transformers and RAGAS. Ray handles parallel execution across large eval batches. The results are served through a FastAPI backend backed by PostgreSQL, with every run logged as an append-only audit record. The frontend is a Next.js 14 app with Tailwind and Framer Motion, deployed on Vercel, including a fully interactive demo page that walks visitors through a simulated evaluation run and a sortable, filterable leaderboard built on the same risk-matrix visual system as the eval reports themselves.

Challenges we ran into

Getting the judge agent to be consistent was harder than getting it to be accurate — the same response scored differently across runs until we added a second judge model for disagreement escalation and fixed the rubric to specific, checkable criteria instead of open-ended quality judgments. Designing the attack taxonomy itself required real restraint: it's easy to write attacks that are technically adversarial but clinically implausible, and we kept having to ask whether a real bad actor or a real edge case would actually produce that prompt. On the product side, making the demo page feel credible without requiring visitors to bring their own API key meant scripting realistic scenario content by hand rather than relying on lorem-ipsum placeholders, which took longer than the UI work around it.

Accomplishments that we're proud of

The risk matrix — the single visual that shows every system against every attack category at a glance — became the project's identity, and it maps directly and legibly onto EU AI Act Articles 9, 12, and 15, which was the whole point: turning an abstract compliance obligation into something a team can actually run in CI. We're also proud that the eval engine is genuinely framework-agnostic on day one, not a LangChain-only tool, because interoperability was the failure mode we most wanted to avoid.

What we learned

Clinical AI safety testing has more in common with security red-teaming than with traditional ML evaluation — the mindset of "how would someone break this" produces a different and more useful test suite than "how accurate is this." We also learned that publishing safety scores responsibly is as much a governance problem as a technical one: score reproducibility, disagreement handling, and protecting against leaderboard gaming needed as much design attention as the model pipeline itself.

What's next for ClinAgent Arena — Open Clinical AI Safety Leaderboard

Next is opening the community submission flow so external teams can publish their own scorecards via pull request, adding demographic bias-delta testing as a first-class scoring dimension rather than a single taxonomy category, and building an auto-generated EU AI Act Article 12 compliance report that a team can export directly from a scorecard. Longer term, we want a live "arena" mode where two systems answer the same case blind and the community votes, producing an Elo-style ranking on top of the fixed benchmark scores.

Built With

Share this project:

Updates