Consistency Mirror

Inspiration

I kept seeing the same Reddit moment over and over: a mod team scales up, two mods quietly start making different calls on the same kind of content, and a week later the community is the one that notices first. Then comes the "the mods are inconsistent" thread, then comes the meta drama. The whole thing is preventable but nobody has the visibility to prevent it.

I read a 2024 paper that analyzed 4.3 million Reddit moderation logs and found that 13.5% of moderation actions are disputed — the "gray area" where reasonable mods just genuinely disagree. And 93% of bot actions in that zone get reversed by humans. That number stuck with me. There's no tool that surfaces this proactively. So I built one.

The whole product is framed around "helps teams align" — not "catches bad mods." That framing decision shaped every UI string in the app.

What it does

Consistency Mirror reads your subreddit's mod log every six hours, computes a semantic embedding for each removed/approved item, and finds pairs of near-duplicate content that received different outcomes from different mods. Those pairs surface on a custom-post dashboard pinned in the sub.

For each pair, the mod team can:

• See both posts side-by-side with similarity score and which mod made which call
• Click "✦ Suggest with AI" to get Gemini's draft of a neutral team rule
• Pick the implied outcome (approve / remove) and save the decision
• Open a discussion thread — the app submits a markdown self-post with both excerpts so the team can debate in-thread

The killer feature is the "Decisions in action" tab. After you log a rule, the system keeps scanning new mod actions. When something semantically matches a logged decision, it gets flagged as ✓ Aligned (action matches the rule) or ⚠ Diverged (action contradicts the rule). The team's accumulated decisions become the living rulebook, applied automatically.

There's also a one-line "Team agreement: 73%" bar and an outcomes breakdown so judges (and mods) get a number to remember.

How we built it

Devvit Web with React 19, Tailwind 4, Vite on the client. Hono on the server with @devvit/web/server for redis/reddit/settings access. Embeddings via Gemini gemini-embedding-001 through generativelanguage.googleapis.com (which is on Devvit's global HTTP allowlist, so no admin approval needed — a small but real win). AI rule drafting via Gemini 2.5 Flash with a tight prompt that constrains the model to "1–2 sentences, no accusations, conditional language."

Redis carries everything — raw moderation actions as sorted sets keyed by timestamp, embeddings cached per action ID so we never re-embed, pair indexes sorted by similarity descending, and the team's logged decisions. The dashboard reads only from Redis so it always renders, even if the cron is mid-flight.

Pair detection is a deterministic O(n²) cosine sweep over up to 200 actions (~20k comparisons, completes in under a second). The threshold is tuned to Gemini's embedding distribution.

A 6-hour cron job runs ingest → embed → analyze. There's also a manual "Refresh" button and a "Seed demo data" mod menu action that injects a corpus of 14 realistic borderline content pairs (self-promo, meta-rants about mods, AI disclosures, low-effort memes, "selling my gear" posts, etc.) so a judge installing on a fresh test sub sees a populated dashboard in one click instead of waiting six hours.

Challenges we ran into

Honestly, the biggest blocker was outbound HTTP. I started with OpenRouter for embeddings because I had credits — turns out openrouter.ai isn't on Devvit's global allowlist and adding it would trigger a Reddit admin approval that could take days. Spent a few hours debugging a "HTTP request to domain X is not allowed" gRPC error before realizing the request had to be re-submitted for review. Pivoted to Gemini because generativelanguage.googleapis.com IS globally allowlisted. Same code shape, different host, demo unblocked.

Same thing happened with the seeding plan — I originally wanted to pull real r/AskReddit content via the public Reddit JSON API. Turns out the schema explicitly forbids adding reddit.com to the HTTP allowlist (it has to go through Devvit's Reddit client, which doesn't expose hot posts from arbitrary subs). So I bundled a hand-written corpus of realistic moderation gray-area content instead. Honestly it's better — every pair is a recognizable moderation scenario.

The other one: the master-detail layout collapsing on narrow iframes. My first pass was a lg:grid-cols-[360px_1fr] two-column layout. The Devvit custom-post iframe is narrower than lg: so it stacked vertically — list above, detail below — meaning clicking a pair sent you to the bottom of the page and the user got lost. Rebuilt it to swap between list-only and detail-only at narrow widths, with a "← Back to all pairs" button. Familiar mobile pattern from Reddit's own app.

Also lost an hour because Devvit's settings store wipes a value any time you change the setting's declaration in devvit.json. Re-set the key three times before realizing.

Accomplishments that we're proud of

• The decision loop actually closes. It's not just "list of disagreements." Log a rule → next matching action gets flagged within minutes. That's the product working as advertised, not as promised.
• AI suggestion feels magical. One click, one second, the textarea fills with a neutral, conditional team rule. The outcome pill auto-selects based on the rule's first word. It's a moment that lands in a 60-second demo.
• The framing. Every UI string says "alignment opportunity" or "two mods, two outcomes." No accusations, no leaderboards. New mods get a way to learn the team's living rules without anyone having to write a wiki page.
• The Pinterest-inspired design language — soft cream surfaces, Pin Sans-substituted typography, single-accent red CTA — keeps the chrome quiet so the content of each disagreement does the talking.

What we learned

A lot about Devvit's runtime constraints — outbound HTTP allowlisting, settings lifecycle, scheduler config in devvit.json, the iframe context limits, how getCurrentUser() can return null even when the user is signed in (had to relax the mod gate to keep the demo working). The Reddit Developer Platform docs are excellent but the rules around what can and can't be allowlisted aren't surfaced obviously — that part took experimenting.

Also learned that the right embedding model and threshold matter way more than the clustering algorithm. I built deterministic agglomerative-style pair detection, but the real lever was tuning the similarity cutoff (0.75 for Gemini, would've been 0.82 for OpenAI). The "is this similar enough to count" call is the whole game.

What's next for Consistency Mirror

• Mod-mail digest. Once a week, the app sends mod-mail summarizing new pairs and the team's drift over time. Mods who don't open the dashboard still benefit.
• Per-pair inline match. When a new action matches a logged decision, post an automod-style note: "Heads up — last week the team decided to allow content like this." Catches divergence at the moment of the call, not 6 hours later.
• Tunable threshold per sub. Some communities are stricter than others — the 0.75 default isn't universal.
• Real "Open Discussion" thread that auto-closes when the decision is logged in-app, so the discussion archive maps cleanly to the rulebook.
• Per-subreddit alignment trend graph — not a leaderboard, just a line showing the team's agreement % moving over weeks. Skipped this round because keeping the UI Reddit-simple mattered more than another chart.

Built With

• cosine-similarity
• devvit
• embeddings
• gemini-api
• gemini-embedding
• google-gemini
• hono
• node.js
• react
• reddit-developer-platform
• redis
• semantic-search
• tailwindcss
• typescript
• vite