dupe-detector

Catch repeat posts before they become moderator work.

Comment

Inspiration

What it does

How we built it

Challenges we ran into

Accomplishments that we're proud of

What we learned

What's next for dupe-detector

Inspiration

Every active subreddit has recurring posts. Some are spam, but most are simply people not knowing what has already been asked. Mods lose time searching and comparing, users feel punished, and older useful threads disappear from the conversation. We wanted a mod tool that catches duplicates without sending subreddit data to a third-party model.

What it does

dupe-detector is a Reddit-native moderation tool for duplicate and recurring posts. It has two workflows:

  1. Poster precheck: a redditor opens "check for similar posts", pastes a draft title/body, and sees the five closest recent posts before submitting.
  2. Moderator duplicate queue: moderators open "duplicates queue" and get a case board of recent posts whose strongest match against the recent subreddit pool crosses the duplicate threshold.

Each match card shows match percentage, shared fingerprint count, author, age, and a link to the original thread. Moderator cards include lock, remove, and leave actions, with server-side moderator checks before any action runs.

How we built it

Devvit Web, React, Hono, Redis, Reddit API, TypeScript, and a deterministic local trigram-cosine matcher. The app uses Reddit APIs for recent subreddit posts, caches recent pools in Devvit Redis for six hours, and runs all scoring locally.

The v1 app intentionally uses no external model or third-party HTTP service. That keeps it cheap, review-friendly, private, and reliable.

Project impact

  • High-volume question communities can catch repeated broad questions before they create another redundant thread.
  • Support/help communities can redirect recurring beginner questions to existing answer threads.
  • Marketplace, deal, and local communities can surface near-identical listings or recurring local questions without manual search.

Impact for moderators: fewer repetitive removals, faster queue review, fewer silent removals that frustrate users, and a cleaner path from repeat post to canonical thread.

Challenges

The hard part was making the tool useful without pretending it knows intent. A duplicate detector should not make the final moderation decision by itself. It should show likely matches, explain why they look similar, and leave the choice to the user or moderator.

Accomplishments

  • Zero external API surface.
  • Two useful surfaces: poster precheck and moderator queue.
  • A distinct case-file / evidence-board UI that makes duplicate review understandable.
  • Minimal permission model: Redis plus Reddit moderator scope.
  • Public proof package includes a live r/AskReddit data screenshot with exact Reddit permalinks.

What's next

  • Per-subreddit threshold tuning.
  • Optional canonical-thread cards for recurring questions.
  • Configurable known-repeat-topic lists for mods.
  • Semantic matching only if moderators and Reddit reviewers ask for it; v1 intentionally stays deterministic and local.

Built With

Share this project:

Updates