Model Scout

Inspiration Featherless's pitch is 21,702 open-source models behind a single API. Two minutes in, that stops being exciting and starts being a problem: which one do I use?

You have three options today. Guess. Trust a public leaderboard that was scored on somebody else's task. Or run a real bake-off yourself — the only correct answer, and it costs an afternoon per decision.

We wanted that third option to take ninety seconds.

What it does You describe your task in plain English. Model Scout then:

pulls the live catalog and narrows 21,702 models down to ~60 real candidates has an LLM shortlist 4 entrants, with a stated reason for each generates test prompts and a weighted rubric from your task — not a generic benchmark runs every model against every prompt, recording latency and token usage grades every answer with a judge model, then runs a head-to-head tiebreak if the top two land within half a point returns a ranked leaderboard — quality score, p50 latency, token cost — plus a copy-paste snippet configured for the winner In our demo run, two models tied at 8.67, so the tiebreak decided it. But the runner-up was 7.8× faster — 7.3 seconds versus 56.9. If a user is waiting, you ship the model that came second. That is precisely the decision a public leaderboard cannot make for you.

How we built it Python 3.11, FastAPI, httpx, Pydantic v2, SQLite, and vanilla JS with no build step. All inference runs on Featherless through its OpenAI-compatible API.

Module boundaries are strict, and that was deliberate:

client.py is the only file that touches HTTP, so the provider is swappable — repointing one environment variable is the entire migration scoring.py is pure functions with no I/O, which is why the ranking logic is fully unit-tested offline Progress streams to the browser as newline-delimited JSON, so the leaderboard fills in live rather than appearing after a spinner. 34 tests run in under a second with no network access.

Challenges we ran into Three, and all of them only appeared against the real API.

The catalog is mostly noise. Our first pool of 60 candidates was full of things like 0xArkad/Qwen2.5-0.5B-Instruct-Gensyn-Swarm-nasty_short_owl. The long tail of the catalog is distributed-training artefacts, LoRA merges and personal checkpoints — useless entrants and wasted credits. We fixed it with a four-pass narrowing: normalise the payload, drop junk/gated/non-chat checkpoints, rank survivors by vendor reputation and parameter count parsed out of the model id, then round-robin across vendor families so the pool is diverse rather than forty Qwen variants.

Effective concurrency was 1, not 4. Featherless meters concurrency in units, and a request's unit cost scales with model size — the free tier's entire budget is 4 units, while a single large-model request costs 4. Our asyncio.Semaphore(4) was generating its own 429s: nvidia/AceInstruct-72B recorded zero completions in an early run purely because our own client had rate-limited it out of the competition. Rather than hardcode a 1, we wrote an AdaptiveLimiter that starts optimistic and permanently shrinks its ceiling the first time the provider reports a concurrency rejection, parsing the 429 body to distinguish that from an ordinary rate limit. asyncio.Semaphore cannot be resized after construction, so it is an explicit counter plus a condition variable.

Open models do not reliably return JSON. Across a heterogeneous catalog you get fenced blocks, Sure! Here's the JSON: preambles, trailing commentary, and "8/10" where a number was requested. A judge that raises on prose cannot grade an open catalog. Parsing degrades through three strategies — strict, fenced, then a brace-balanced scan that correctly ignores braces inside string literals — and reports failure instead of throwing.

Accomplishments that we're proud of Failure is treated as data. A model that 404s, times out, or turns out to be gated stays on the leaderboard with its error and a dash — never a fabricated zero — and the run degrades around it. In the demo, openchat was down with an HTTP 503 and recorded 0/3 while the bake-off completed normally. There is a test asserting that a model with zero completions can never outrank one that produced output.

We fit the whole thing inside 25 inference credits. Rubric scoring instead of a pairwise round-robin (one judge call per generation, rather than ten pairs per prompt), a tiebreak restricted to the top two and only when they're genuinely close, and SQLite caching keyed on a hash of the request.

What we learned Most of the interesting engineering lived in the gap between what the docs say and what the API does. Concurrency units, cold-start latency, catalog composition, JSON compliance — we discovered all of it by running against production and reading error bodies carefully rather than by reading documentation.

We also learned that a cache can silently corrupt your results. Our first caching implementation replayed every run reporting 0 ms latency, quietly deleting the most valuable column on the leaderboard — and it did so without failing any test. Cached rows now store the latency the call was originally measured at.

What's next for Model Scout Larger eval sets with confidence intervals, so the leaderboard becomes statistically defensible rather than directional Multi-judge ensembles to reduce single-judge bias, including self-preference bias toward the judge's own model family A CI export, so a team can re-run its model choice on a schedule and get alerted when something better or cheaper appears in the catalog Cost modelling against real per-model pricing rather than a blended token estimate