Inspiration
Popular LLM benchmarks are useful, but many rely on static question sets that have been publicly available for years. Those questions may appear in model training data, so a high score can sometimes reflect memorization instead of the capability being measured.
I wanted a simple way to benchmark the local models I actually run, using questions that I control and can regularly replace. That idea became FreshBench: a local-first benchmarking tool built around private question suites, deterministic grading, and reproducible results.
What it does
FreshBench is a fully terminal-based benchmark tool for local language models exposed through an OpenAI-compatible API.
Users can create their own benchmark suites, keep the real questions outside Git, and refresh them over time. Public example questions are included only to document the format and let new users test the application immediately.
FreshBench provides both a command-line interface and an interactive TUI. It can:
- Run one or multiple benchmark suites
- Grade responses using deterministic Python-based graders
- Measure latency, time to first token, throughput, token usage, and retries
- Display model reasoning separately when supported by the endpoint
- Save incremental checkpoints and resume interrupted runs
- Compare results only when suite fingerprints and configurations are compatible
- Diagnose endpoint capabilities such as streaming, tool calling, usage data, and tokenization
The current graders support exact text, accepted answers, substring requirements, regular expressions, numeric values, JSON, JSON Schema, and structured tool calls.
FreshBench intentionally does not use an LLM as a judge. Strict pass/fail is always the primary score, while deterministic partial scores are shown only as secondary diagnostics.
How I built it
FreshBench is written in Python and packaged with Astral's uv. It is published on PyPI and can be installed as a global CLI or launched temporarily with uvx.
The interactive interface is built with Textual, while Typer and Rich power the non-interactive CLI and terminal reporting.
The model client communicates with OpenAI-compatible Chat Completions endpoints, making it compatible with local tools such as llama-server and llama-swap. It supports streaming responses, reconstructs fragmented content and tool calls, and automatically falls back to non-streaming requests when necessary.
Benchmark suites are written in versioned YAML files. Before a run begins, FreshBench validates every question, grader, regular expression, JSON Schema, and tool definition.
Each suite receives a content fingerprint. Those fingerprints are stored in result files and checkpoints, preventing results from changed question sets from being compared or resumed accidentally.
The application keeps transport, suite loading, execution, grading, persistence, diagnostics, and presentation separated so each part remains small and understandable.
I used Codex throughout development to help inspect the codebase, plan changes, implement features, debug issues, write tests, and repeatedly validate the complete application.
Challenges
The hardest part was making benchmark runs both resilient and reproducible.
OpenAI-compatible endpoints are not completely identical. Some support streaming, usage metadata, reasoning fields, native timing information, tool calls, or tokenization routes, while others support only a subset. FreshBench needed to handle those differences without becoming tied to one model server.
Streaming was another major challenge. Tool-call names and arguments can arrive across multiple chunks, and reasoning models may return visible output and reasoning through different fields. The client must reconstruct these correctly while also measuring time to first token and stream stalls.
Checkpointing and resume safety also required careful design. Simply skipping completed questions is not enough: the model configuration, generation settings, suite names, fingerprints, and question limits must still match. Otherwise, the final result could silently combine incompatible runs.
The TUI also had to present a large amount of information without becoming overwhelming. It needed to show live progress, scores, timing information, category summaries, errors, retries, reasoning, output, and grader details while remaining usable in a normal terminal window.
What I learned
This project taught me that benchmarking is not only about sending prompts and counting correct answers.
A useful benchmark must clearly define:
- What is being measured
- How answers are graded
- Whether results are comparable
- How failures and retries are recorded
- Which endpoint capabilities were available
- Whether the question set may have been exposed to the model
I also learned how inconsistent OpenAI-compatible implementations can be in practice, especially around streaming, usage data, reasoning fields, and tool calls.
Most importantly, I learned that freshness alone does not guarantee a perfect benchmark. FreshBench cannot prove that a private question has never appeared in training data. Instead, it gives users practical tools to reduce that risk by owning, protecting, versioning, fingerprinting, and regularly replacing their benchmark questions.
FreshBench is currently alpha software, but it already provides a complete local workflow for creating private suites, running models, inspecting failures, resuming interrupted evaluations, and comparing compatible results.
Log in or sign up for Devpost to join the conversation.