Inspiration
We just kept seeing it that every time you ask a model a question, she does it right, then you write it down, then you reword the same question and the answer changes. One-shot evaluation gives brittle models a sense of reliability, and benchmark contamination gives memorization a sense of understanding. All this is concealed in a single number, the accuracy of the leaderboards. We wanted an evaluation that isn't asking a model a single question, it's asking him all the questions and reporting back how much they think they can really trust the answer from this model.
What it does
EvalStorm tests a language model and coeliability score that you can justify. Rather than a single prompt for each question, it has a barrage of prompts over four stages: 1. Paraphrase Storm — restates each question the answer stops on.
- Bloom Ladder — explores the same topic in each of Bloom's taxonomy (remember → understand → apply → analyze) to distinguish between memorizing and understanding. 3. Concept Web — tests the fundamental conceptsy.
- Hierarchical Judge + Bias Guard — grades correctness, completeness, clarity, and instruction-following on an established rubric, shuffles the order of the prompts and uses anonymized prompts to reduce position/verbosity bias.
It outputs the accuracy for the baseline (single-prompt) and the workflow composite, and — the most important number — an instability index: the lower, the more stable the model is across phrasings; the higher, the more that the answers vary with phrasing. It also becomes a sign of a possible contamination pattern when a Remember-level scor collapse. All runs are in the Demo Mode (no API key, all numbers are fully deterministic, no costs) or Live Mode (true provider calls, no costs), and all numbers export to CSV/Excel, Markdown, HTML or raw JSON which can be re-derived.
How we built it
Features: Python + FastAPI; 1 daemon thread per run; JSON-per-run persistence; and a disk cache that means that the same prompt is never paid for twice (LLMeBench pattern). There are two types of provider abstraction: a single OpenAI-compatible across Groq, Featherless, OpenCode Zen and local Ollama/llama.cpp, and a deterministic simulated backend for the demo. The only environment variable that shouldn't be moved is swapping providers. No code changes, change only.
- Frontend: React + Vite dashboard — run creation, live polling progress, run results dashboard, detailed per-question dashboard evidence, and export.
- Methodology: based on published evaluation research that is grounded in evaluation that is published and relevant to the field; such as, SciEx (EMNLP'24), StructEval (ACL Findings'24), HD-Eval (ACL'24), LLMeBench (EACL'24), ConSiDERS (ACL'ion (TACL'24).
- Deployment: Render (FastAPI API + static frontend) and keep alive with UptimeRobot.
Challenges we ran into
The free tier rate limits were extremely harsh. On token-per-minute 429s, Groq was unable to keep up and Featherless limited us to 4 concurrency. All Strong models have a cost of 4 units, and the free stealth models from OpenCode Zen flap in and out ("upstream endpoint unavailable"). In the backend we added a guard for and hardy with Retry-After-aware backoff. gateways that send back an HTTP 200 containing an error bo Returning messy JSON is an issue with reasoning models. A model that consumes its budget on the token count while emitting its JSON will still be able to be parsed.A model that exhausts its token consumption budget while it is emitting its JSON will be truncated but still be able to be parsed. As the wrong shape — as a form that is misshapen. Added shape guard tad to avoid pipeline crash. Maintaining the trustworthiness of the demo. The simulated backend needed to be deterministic and distinctly labeled for the judges to be able to recreate the exact run, and still present a real (not zero) instability index.
- Security discipline. The provider keys are read posed to the frontend/committed.
Accomplishments that we're proud of
- No live-API roulette on camera, deterministic Demo Mode was completed and eveeal reliability story was finished.
- Real Live Mode runs completing the full 32-qagainst a real provider, no fallbacks orhardcoding. The instability index is a single number that is completely obscure. An agnostic architecture that has withstood three fires from provider changes.
- Exports - every number is a direct link to the raw run JSON - no numbers faked.
What we learned
Prompt brittleness is a reality, it's large, and it's measurable: The difference between the baseline accuracy of the model and its stability when it comes to cross paraphrasing is often the most genuine commentary you can make about the model. Evaluation is largely about infrastructure. The "AIt is" part is about retries, caching, rate-limit handling, and defence of flaky free-tier endpoints. With a clean provider abstraction, you can replace anything under you! Our metrics were not arbitrary, but grounded in the published methodology.
What will happen to Eval Storm?
- Multi-model comparison — compare several modall's W agreement (this is already anticipated in the code).
- Stage 0: Fresh Test Material — comes up with questions no one has encountered before to completely outflank benchmark contamination.
- Statistical confidence: confidence interval, significance testing of the instability index. Broadened benchmarks – larger number of domains, challenging levels, increased number of questions.
- Contamination detection — make the c a trusted signal.
Log in or sign up for Devpost to join the conversation.