Meme's on main page
Meme's page
Main page as seen on devme.me open

DevMeme: Learning if any of Twelve Frontier Models can make a Joke

DevMeme ingests developer memes, asks twelve frontier models to describe, classify, explain, and riff on them, then serves the results through a fast browsing site. The research question is simple: can a model be funny, and how would we measure that?

Inspiration

I started with a suspicion I could not shake:

On Reddit, the meme is the setup. The comment section is the punchline.

You open a meme about a Friday production deploy. It earns a mild smile. Then you reach a comment comparing the 4:59 p.m. pager alert to a jump scare. That is the line you screenshot and send to your team. The image set it up; the thread made it funny.

That creates a hard problem for an LLM meme understander. Much of the humor lives in shared context: war stories, team rituals, and the experience of engineers who have all kubectl apply-ed their way into the same crater. A caption model can describe "a stressed developer at three monitors" perfectly and still miss the joke.

I wanted DevMeme to do two jobs. The practical job is a fast, richly searchable library of developer memes at devme.me. The research job uses that library as a testbed for model humor and for a benchmark based on actual preferences.

What it does

DevMeme has a content pipeline and a public site.

The pipeline scrapes media from Telegram tech channels and builds several layers of metadata:

Visual description: the scene, layout, visible text, and technical setup.
Humor explanation: the shared developer experience that makes the image work.
A tech-savvy joke: each model's attempt to add a comment worth reading.
Three-tier tags: categories → common tags → context tags. Controlled categories and common tags keep search out of the synonym swamp; context tags retain the details.
Deep dives: 2,000 to 8,000-character explanations that progress from ELI5 through Junior, Senior, and Architect levels. A race-condition meme can also teach race conditions.

The site uses Astro SSR with Preact islands and no React dependency. Preact ships roughly 3 KB over the wire (minified and gzipped), compared with roughly 45 KB for React plus ReactDOM. That difference matters on a rapid-fire browsing surface.

There are many ways in: a search bar that queries the full corpus as you type, a Latest feed, category and tag hubs, a Random jump, and Previous/Next paging with a position counter on every meme page. On desktop, J and K drive that paging from the keyboard: go ahead, mash them, and enjoy what Preact plus prefetching feels like <3. The mobile version uses swipe gestures instead. Each device lands on its native version automatically: phones get the mobile app, larger screens get the desktop layout, and crawlers see both as separate indexable URL families. Phone users who prefer the full layout can switch through "Open desktop site" in profile settings, and the choice sticks for six months.

Browsing hubs live at meme categories and tech tags, and hovering any tile previews real memes from that facet before you commit to the click. The hover previews are a personal favorite: once I figured out how they could work, Codex running GPT-5.6 Sol built them. The Discover feed runs on hidden-gems logic: a seeded rotation that picks a couple of memes per tag across the whole taxonomy and reshuffles every two hours, so deep cuts keep resurfacing.

Every same-origin link starts downloading its page the moment it appears, even off-screen. By the time you click, the page already sits in your browser cache, so the click skips the network entirely.

Twelve model voices in the corpus

The corpus accumulated model outputs from several runners and processing periods, so coverage varies by meme and run.

Family	Models
Anthropic	Claude Opus 4.1, Claude Opus 4.6, Claude Sonnet 4.5, Claude Fable 5
OpenAI	GPT-5, GPT-5-Pro, GPT-5.5 (xhigh), GPT-5.6 Sol (max), O3-Pro, o3-deep-research
xAI	Grok 4.1
Google	Gemini 3

Eleven of the twelve are classifier voices; o3-deep-research writes deep dives only. The six longest-running voices each cover roughly six thousand memes, Gemini 3 covers about 4,800, and the newest voices (Claude Fable 5, GPT-5.5, Claude Opus 4.6, GPT-5.6 Sol) cover several hundred to a thousand each. Backfill continues where it stays affordable; a full backfill across every voice is expensive enough that some of it will probably never happen.

The models arrived through two doors. The API pipeline calls Azure, Bedrock, and OpenRouter endpoints directly. The agent harnesses (Claude Code for Claude Fable 5, Codex for GPT-5.6 Sol, Jules for Gemini 3, and both routes for GPT-5.5) let a coding agent open each image itself, search the web for temporal context around the post date, and write the same JSON files by hand.

Deep dives come from four of those voices. GPT-5.5, Claude Fable 5, o3-deep-research, and GPT-5.6 Sol have all written long-form explanations, and when a meme has several, the site picks one in that priority order. An older generation is still on stage too: GPT-4o, GPT-4.1, and Claude 3.7 Sonnet jokes run on the site alongside the newer voices, and they double as the benchmark's historical baseline.

Tags tend to converge. Jokes scatter. Grok may reach for a roast while Opus favors a wry observation and GPT builds a more structured line. Every model answers the identical joke prompt on purpose, so the spread comes from the model, never from prompt tuning. That spread is useful data because humor behaves like a taste distribution.

Measuring humor

The Reddit pattern points at a benchmark built on comparison, and the comparison is deliberately simple: the votes developers already cast.

Step 1: The joke is in the comment

The image is the setup; the line you actually screenshot usually lives in the comment thread beneath it. So the benchmark ignores plain captioning and targets the riff, the war story, the reply that lands. Each model writes a joke on every meme, and those model jokes sit on the same page as the real human comments imported from the meme's Telegram thread. The model's job is to be a good commenter.

Step 2: Developers vote

DevMeme already lets users like or dislike every joke while browsing, model-written jokes and imported human comments alike, on the same buttons. That is the whole instrument. The votes are self-selected and exposure-dependent, so I treat them as a behavioral preference signal from participating developers, not a lab result.

Step 3: Count votes per category, up and down apart

No rating model, and no LLM jury. The ranking reads straight off the tallies, with two deliberate rules:

Up and down votes stay separate. A joke with 100 likes and 90 dislikes is not the same as one with 10 likes and no dislikes, so approval and rejection stay two numbers instead of one net score.
Scores are matched per category. A model can be funnier about frontend than about on-call trauma, so totals are compared category by category, not in one lump.

The question each comparison answers: in a given category, do a model's jokes earn more developer approval than the human comments do? Humor here gets judged only by the people it is for.

One deliberate constraint: the votes must stay blind. Knowing which model wrote a joke changes how people vote on it, so joke-to-model attribution stays sealed for now. The benchmark results go public once jokes collect 5,000 combined like and dislike votes, or on August 28, whichever comes sooner. Until then, every like or dislike you cast counts twice: it immediately moves how jokes rank on devme.me, and the same vote is a ballot in the sealed benchmark.

How we built it

The ingestion pipeline lives in its own repository upstream of this site. The short version: every enabled model gets its own image queue with bounded retries, a controlled vocabulary (102 categories, 2,799 common tags) keeps tags joinable across models, prompt caching keeps twelve-model costs sane, and every original model output stays on disk as provenance.

Astro SSR calls services directly. Meilisearch supplies meme cards, detail lookup, search, categories, tags, and feeds. MongoDB through Prisma stores accounts, bookmarks, votes, reader comments, analytics, communities, and other user state. Redis owns the shared HTML cache plus live session, rate-limit, view, and trending state across exactly four clustered devmeme-web workers.

Redis is sized for the full public corpus of roughly 15,300 pages, and every deploy re-warms all of them, failing the deploy under 90 percent verified cache hits. Prefetch dedupe expires on the same 300-second clock as the browser page cache, and a background scan re-warms links that expired during long sessions.

Repository measurements from July 19, 2026 recorded up to 728 requests per second on the warm path and a cold-SSR ceiling near 37 requests per second. Those figures are dated engineering measurements, not a service-level objective.

Codex served as the engineering harness for GPT-5.6 Sol, which pulled double duty: it ran its own classification and deep-dive batches as a corpus voice, and it reconciled the path from downloaded assets to MongoDB, Meilisearch, and the public site. It also finished desktop and compact flows, hardened search and delivery, and browser-verified the demo.

Challenges we faced

A joke has no gold label. Like and dislike counts are an honest signal, but they stay self-selected and exposure-dependent, so the ranking only means as much as the turnout behind it.
The image supplies only part of the context. Teaching a model to write the comment, with the war story and timing a developer recognizes, remains the harder prompt problem.
Providers disagree on authentication, API shape, rate limits, JSON behavior, and failure modes. Large batches expose all of those differences. Per-model queues, bounded attempts, logging, and incremental processing keep one endpoint from stopping the corpus.
Prefetch failures appeared in long production sessions that short local checks missed. Viewport gating, low request priority, and permanent dedupe each broke the browsing flow. Browser cache expiry and prefetch dedupe now share the same lifetime.
The cost multiplies quickly across model calls and long deep dives. Prompt caching and skipping completed outputs keep the pipeline economically usable.

What we learned

Agreement and disagreement answer different questions. Converging tags improve search consistency. Diverging jokes create the comparisons a humor benchmark needs.

The voting interface can double as research instrumentation, although votes from participating users carry selection and exposure effects. Product behavior supplies useful evidence only when the analysis respects those limits.

Keeping up-votes and down-votes as separate numbers, and splitting scores by category, tells you more than one blended number would: where a model actually lands, not just whether it wins.

Fast browsing also taught me to treat performance as a coupled system. Prefetch timing, browser cache lifetime, Redis residency, and deploy-time warming have to agree. One stale assumption becomes a pause the user can feel.

What's next

The big unsolved one: exposure. The entire vote-driven design lives or dies on developers showing up, and today the site has not enough of them. Distribution comes before everything else on this list; the 5,000-ballot clock only ticks when people vote.
Publish the benchmark results the moment the blind embargo lifts: 5,000 combined votes or August 28, whichever arrives first.
Add new voices as frontier models ship. The recently released Kimi K3 is first in line, and every newcomer answers the same frozen joke prompt, so its scores land on the same footing as the original twelve.
Ship the head-to-head mode: two jokes for the same meme, side by side, pick the one that lands. The imported human comments already compete in the vote pool today.
Ship the native iOS app. A pocket-native swipe feed is the natural home for this kind of browsing.
Ship the native Android app. Every new surface feeds the vote pool.
Ship on Apple Vision Pro and whatever glasses follow it. A room-scale wall of developer memes serves no research purpose whatsoever; I want it just for the memes.
Open contribution: submit your own meme, and once it clears review it joins the library.
Calculate category-specific ratings to see whether model performance changes across frontend snark, management jokes, and operations trauma.
Further out: communities, so people can organize and curate their own collections.

DevMeme is already a searchable, fast-browsing library for developer humor. The research layer turns it into a direct test: can a frontier model out-riff the best human comment on the same meme?