MonsterBench

Enterprise LLM Benchmarking Suite. Create an organization, invite members, benchmark inference configurations, and analyze runs.

Comment

Inspiration

Picking an LLM for a real task means juggling a dozen models with different prices, speeds and quirks. Most teams pick one on vibes, or compare a handful of prompts in a notebook that nobody can reproduce later. We wanted a tool where you bring your own data, from a file, a bucket or your database, and get a repeatable answer to the question: which model and which prompt actually works best for my task, at what cost? Then you can ask follow-up questions about the results in plain English, or out loud.

What it does

Brings in your records, wherever they live

  • Upload CSV, JSONL or JSON datasets. BenchMonster infers a typed schema for each, which you can edit and validate.
  • Connect your own storage: S3-compatible object storage (Vultr Object Storage, AWS S3, R2) or Postgres (Tiger Cloud, Supabase, RDS). Each connection shows whether it's read-only or read-and-write, verified by a real access check.
  • Stream large tables straight from Postgres without copying them. Runs page through the records in chunks and save their position, so a server restart resumes the run instead of losing it.
  • Filter records (equals, one of, contains, ranges, regex), dedupe, or take a repeatable random sample. For database tables, this runs inside the database.
  • Choose which fields of each record reach the prompt, rename them, and parse each one (integer, number rounded to N decimals, boolean, JSON), with a live preview.

Configures and runs inference

  • Benchmark profiles: a prompt template, where each {{variable}} comes from a record field, the whole record, an entire dataset, or fixed text; plus expected outputs, output processing, scoring and models.
  • Scoring methods: exact match, contains, regex, numeric tolerance, JSON Schema validity, field-by-field JSON comparison, and LLM-as-judge. Output processing (regex or JSON-field extraction) runs before comparison.
  • Run modes: realtime (one streaming request per record, measuring latency and time to first token) or batch (many records packed into one request to cut cost).
  • Metrics per model: accuracy, pass rate, p50 and p95 latency, time to first token, throughput, tokens (including reasoning tokens), total cost and cost per correct answer.
  • Exports: send results to your own bucket or database, manually or automatically after every run.

Analysis tools (BMQuery)

  • A read-only SQL console over your organization's benchmark data, or directly over any connected database. It has saved queries and CSV or JSON download.
  • An AI analyst that writes the SQL for you, labels which database each query is for, and fixes queries when you send it the error. It can look up runs, profiles, columns and sample records to get names right. It never mixes two databases in one query; that rule is enforced by a server-side check.
  • Voice mode: push to talk, and replies can be read aloud.
  • Organization memory, so the analyst recalls findings from earlier sessions.

Versioning and reproducibility

  • Every edit to a profile creates an immutable version, tagged with what changed (prompt, records, scoring, models, settings). You can restore or duplicate any version.
  • Prompt library versioning: pick a prompt and a version when building a profile. BenchMonster tracks which prompt version each profile used, and flags when a newer one exists.
  • Every run records exactly which profile version, prompt version, settings and records it used.

Teams: organizations with owner and admin roles and invite links. Each organization's data is isolated down to the database level.

How I built it

  • Frontend: Next.js 16 (App Router, React 19, React Compiler) and Tailwind CSS v4.
  • Backend: FastAPI (async Python) with asyncpg, plus a custom concurrent runner with per-model rate limits, retries and streaming.
  • Database: Tiger Data (TimescaleDB on Postgres). Results are stored in a hypertable. Each organization gets its own schema of pre-filtered views and its own read-only login, so the SQL console and the AI analyst physically can't see another organization's data.
  • Model inference: Vultr Serverless Inference. It's OpenAI-compatible, with 15 open-weight models (DeepSeek, GLM, Qwen, MiMo, Nemotron, MiniMax and more) and per-token pricing read live from Vultr's model catalog.
  • Analyst model: GLM 5.3 on Vultr, with tool calling for lookups and query writing.
  • Memory: Backboard.io, which stores durable findings per organization.
  • Voice: ElevenLabs: Scribe for speech-to-text and Flash v2.5 for text-to-speech.
  • Connections: boto3 for S3-compatible storage, and asyncpg for Postgres with read-only transactions. Saved credentials are encrypted, and hosts that resolve to private or internal addresses are blocked.
  • Deployment: Docker Compose and Caddy (automatic HTTPS) on a Vultr cloud VM, at benchmonster.tech (.tech domain).
  • Testing: pytest, plus Playwright-driven browser tests for every feature before deploying. We built with Claude Code as a pair programmer.

Challenges I ran into

  • Large datasets without copying them. Our first version copied every record into BenchMonster, which duplicated the data twice (once on import and again in every rendered prompt). We rebuilt runs to stream records in chunks from Postgres: filters and random sampling run in SQL, runs are capped at the highest key present when they start (so records added mid-run are excluded), and they resume after a restart.
  • Real multi-tenant SQL. Letting users, and an AI, run arbitrary SQL safely meant a separate schema and read-only login per organization, forced read-only transactions, statement timeouts, and blocking multi-statement queries.
  • Postgres permission edge cases. Listing tables failed on a shared Tiger database because of how name-based privilege checks behave on schemas the login can't access. We switched to checks by internal table ID.
  • Keeping the AI analyst honest. It tried to join tables across two separate databases, and at one point imitated our internal note format instead of calling its tool. We fixed this with a hard rule, a server-side check that sends violating queries back, and replaying its earlier work as proper tool calls.
  • Batch inference on a platform without a batch API. We pack many records into one request, parse the JSON-array answer, and split tokens and cost fairly across the batch.
  • Reproducibility everywhere: immutable versions of profiles and prompts, while still letting people edit freely.

Accomplishments that I'm proud of

  • An end-to-end workflow, from raw records in your own cloud to a versioned benchmark to per-model metrics to analysis in plain English, deployed and working at benchmonster.tech.
  • It surfaced a finding in a real dataset. On a 43,744-record personality survey, BenchMonster showed that the "Introversion Score" is reversed: high scores mean extraverted. By iterating the prompt through versioned profiles, accuracy on the same respondents went from 6.8% (about chance) to 36.5% (adding a per-type guide) to 60% (rules in the system prompt), and we could see exactly which letter the model still gets wrong.
  • The streaming runner reads large Postgres tables directly, survives restarts mid-run, and never duplicates data.
  • Strong isolation: separate views and logins per organization, encrypted credentials, and a check that stops connections from reaching internal addresses (verified on production).
  • Per-feature commits with a browser test before every deploy.

What I learned

  • A good evaluation is mostly about the data, not the model. Knowing what a field actually means (like a reversed scale) moved accuracy more than switching models.
  • System prompts and templates do different jobs. Putting fixed rules in the system prompt measurably improved how well models followed them.
  • Versioning everything (profiles, prompts, run settings) is what turns a benchmark from a one-off demo into something you can trust and compare over time.
  • AI assistants need guardrails enforced in code, not just in their instructions.
  • Postgres and TimescaleDB are powerful enough to serve as the analytics store, the multi-tenant security boundary and the query engine all at once.

What's next for BenchMonster

  • Benchmark the analyst itself: run a text-to-SQL test set through BenchMonster to pick the best and cheapest model for BMQuery.
  • Streaming from object storage: process large CSV and JSONL files in chunks of records, like Postgres tables today.
  • Provider batch APIs (asynchronous jobs at about 50% lower cost) and more providers beyond Vultr, including your own fine-tuned models.
  • Scheduled regression runs, to alert you when a model update or a prompt edit drops accuracy.
  • Side-by-side diffs between prompt versions and between runs.
  • Charts and shareable reports generated from BMQuery results.
Share this project:

Updates

Submission history