I am speed - fast dot com style but for LLMs

iamspeed.dev - fast dot com style but for LLMs. Now with race mode, leaderboard, and more.

Comment

Inspiration

Every developer choosing an LLM eventually asks the same question: which one is actually fast? Academic benchmarks exist but they're hard to reproduce, disconnected from real network conditions, and don't reflect the API you're actually going to call. I built iamspeed.dev to answer that question directly: measure real inference speed against real endpoints, with your own API key, right in the browser. No proxy, no server in the middle, no numbers you can't trust.

What it does

iamspeed.dev is a full-stack LLM inference speed benchmark. Think fast.com, but for language models. Users run real streaming requests against major LLM APIs (OpenAI, Anthropic, Groq, Cerebras, Fireworks AI, Mistral, OpenRouter, Google Gemini, xAI, and local models via Ollama/LM Studio) using their own API keys, and the app measures:

  • TTFT: Time to first token (perceived responsiveness)
  • TPS: Tokens per second (sustained throughput)
  • TTLT: Time to last token (total wall-clock latency)

Results can be submitted to a Community Leaderboard backed by Amazon DynamoDB, giving everyone a shared, real-world picture of which providers are actually fastest.

The standout feature is Race Mode: up to 3 providers stream in parallel on an animated Cars-themed racetrack. Winner is determined by TTLT (with TPS as tiebreaker), plus a "Fastest Start" award for the lowest TTFT.

The frontend is deployed on Vercel. API routes (leaderboard reads/writes) run as Vercel serverless functions and communicate directly with DynamoDB using the AWS SDK v3.

DynamoDB was a deliberate choice for scale. The leaderboard schema is designed to handle millions of submissions without any infrastructure changes: single-digit millisecond reads at any throughput, GSIs that fan out to provider-specific and device-specific views, and on-demand capacity mode so the table absorbs viral traffic spikes automatically. If iamspeed.dev hits the front page of Hacker News tomorrow, the database doesn't blink.

How we built it

The stack is TypeScript + Astro with the @astrojs/vercel adapter for SSR. The homepage and most of the UI are statically prerendered (fast, no cold start). Only the leaderboard API routes and the leaderboard page use on-demand rendering, which maps naturally to Vercel serverless functions.

Each LLM provider gets a small adapter in src/lib/providers/ that normalizes their streaming SSE or fetch API to a shared interface. Most OpenAI-compatible providers share a single generic adapter; only Anthropic and a few others need their own logic.

Race Mode is the heart of the app: up to 3 providers get the same prompt and race in parallel. An animated Cars-themed racetrack updates live as tokens stream in. Winner is decided by TTLT, tiebroken by TPS, and a separate "Fastest Start" award goes to whoever gets the first token out.

The Community Leaderboard is backed by Amazon DynamoDB with two tables:

  • simple_leaderboard: stores submissions (provider, model, TPS, TTFT, TTLT, nickname, device hash, timestamp) with GSIs for top-N global ranking, per-provider filtering, and per-device history.
  • leaderboard_nonces: single-use tokens with a TTL for auto-cleanup to enforce anti-replay.

The table uses on-demand capacity mode, which means it scales automatically from zero to millions of reads and writes per second with no capacity planning. There's no database to resize, no cluster to rebalance, and no ops page to watch at 2am. At 1 million submissions the table just... works, and you only pay for what you actually use.

Submissions are protected by Cloudflare Turnstile (bot prevention), HMAC-signed single-use tokens (anti-replay), device fingerprinting (soft rate limiting), and server-side sanity validation (plausibility bounds on the reported numbers). Because the benchmark runs entirely in the user's browser with their own API key, there's no way to re-run server-side, so we're upfront about the self-reported nature and layer in pragmatic protections instead.

Browser (User's machine)
  │  Streaming SSE/fetch with user's own API key
  ▼
LLM Provider APIs (OpenAI, Anthropic, Groq, Cerebras, etc.)
  │  (never proxied - direct browser to API)
  │
  ├─ Benchmark result ──► POST /api/leaderboard/token  ──► Vercel Serverless Fn
  │                           (HMAC-sign, nonce issue)         │
  │                                                            ▼
  │                                                       Amazon DynamoDB
  │                                                       leaderboard_nonces
  │
  └─ Signed token ──────► POST /api/leaderboard/submit ──► Vercel Serverless Fn
                              (verify HMAC, nonce,            │
                               sanity-check metrics,          ▼
                               Turnstile)                Amazon DynamoDB
                                                         simple_leaderboard
                                                              │
GET /leaderboard page ◄── GET /api/leaderboard ◄─────────────┘
(Preact component,         (top-N, provider filter,
 client:load)               GSI query)

Challenges we ran into

  • Mixed-content blocking: the app runs over HTTPS but local Ollama runs over HTTP. Worked out a Cloudflare tunnel workaround so power users can still benchmark local models from the production site.
  • Token counting accuracy: SSE doesn't have a standard token count. Built a hybrid approach using chunk counting plus provider usage field reconciliation.
  • Leaderboard trust without re-running: the answer is layered defenses (Turnstile + HMAC + rate limits + sanity checks) plus honest copy. Not bulletproof, but solid enough for a community tool.

Accomplishments that we're proud of

Getting Race Mode working smoothly was the one that felt genuinely hard. Coordinating three live streaming responses from three different APIs, updating a shared animated racetrack in real time, handling one stream finishing while the others are still running, and doing all of it without the UI stuttering, that took real work. When it clicked and the cars actually raced, it was a great moment.

The DynamoDB leaderboard architecture is something we're proud of too. It's not just "we used DynamoDB because the hackathon required it." The schema was designed specifically to stay fast at any scale: GSIs that make top-N and provider-filtered queries O(1) regardless of table size, on-demand capacity that absorbs traffic spikes with no config, and TTL-based nonce cleanup with zero application logic. The same table that works today would handle a million submissions without changing a line of infrastructure code.

On the trust layer: we're proud that we were honest about what it can and can't do rather than overselling it. HMAC tokens, Turnstile, rate limiting, and device fingerprinting together raise the bar significantly for abuse. But we added a "self-reported" disclaimer in the UI instead of pretending the leaderboard is tamper-proof. We think that's the right call for a community tool.

Finally, 10+ LLM providers all working through a clean, shared adapter interface. Adding a new OpenAI-compatible provider is about 10 lines of config. That extensibility was by design.

What we learned

  • Token delivery varies a lot across providers. Some stream tokens one at a time; others batch them in large chunks. Getting accurate counts means normalizing across both patterns and falling back to the provider's reported usage field when it's available.
  • DynamoDB's HTTP-based model is a great fit for Vercel serverless. No connection pool to warm up, no VPC to wire up, and cold starts don't hurt you.
  • Coordinating 3 parallel streaming fetches with shared cancellation via AbortController is trickier than it sounds. One dropped stream can't be allowed to stall the others.
  • A tamper-resistant leaderboard is not the same as a tamper-proof one. HMAC tokens stop replay attacks and spoofed submissions, but a user can still manipulate their own numbers in their own browser. Being honest about that in the UI is the right call.

What's next for I am speed

  • Race Mode leaderboard: the submission and trust infrastructure already exists for Simple mode. Extending it to Race Mode is the next obvious step, storing head-to-head results and letting the community see which provider consistently wins.
  • Speed Grade verdicts: stamp every result with a fun tier based on TPS thresholds (think Kachow! for 1000+ tok/s, Ludicrous Speed, Cruising, Rush-hour traffic). Makes results instantly screenshot-worthy and on-brand with the Cars theme.
  • Shareable result image cards: generate a 1200x630 branded PNG client-side via Canvas, with the podium, TPS number, Speed Grade, and model names. Images get 10x the social engagement of text links.
  • Multi-run statistical mode: single runs are noisy. A "Run x5 / x10" option that reports median, mean, and p95 TPS and TTFT turns a fun demo into numbers people actually trust enough to cite.
  • Cost metrics: surface $/request and $/1M tokens alongside TPS so users can compare value, not just speed. Add a "Best Value" award to Race Mode.
  • Tournament / bracket mode: seed 4-8 models into a knockout bracket, run head-to-head rounds, crown a champion. Extremely shareable and a natural extension of Race Mode.

Built With

Share this project:

Updates