• 

  SignalDesk AI - Dashboard

• 

  SignalDesk AI - Dashboard (Mobile)

SignalDesk AI

The problem no one talks about

Reddit moderators don't have an automation problem — they have a signal problem. When a coordinated harassment campaign hits, the modqueue fills with 40+ individual reports. Each one looks identical. Mods click through them one by one, three of them simultaneously, duplicating work and still missing the pattern.

SignalDesk AI was built to fix that.

What it does

SignalDesk AI runs inside Reddit as a native Devvit app. It monitors subreddit activity continuously and detects three classes of incident:

• Report Storm — one post/comment receives ≥5 reports within 30 minutes
• Thread Meltdown — ≥10 reports spread across ≥3 items in the same thread
• AutoMod Flood — ≥8 items filtered by the same AutoMod rule within 30 minutes

When a pattern is detected, it becomes a case — a structured incident object with a severity score (LOW → CRITICAL), a list of affected items, report reason breakdown, and a full AI incident brief generated by GPT-5. Instead of 40 individual queue items, moderators see 3 grouped cases.

The dashboard

Cases appear on a 5-column kanban board (Urgent → Needs Review → Claimed → Actioned → Escalated) hosted on Cloudflare Pages and linked from the subreddit mod menu. A metrics strip at the top shows active cases, items grouped, AI briefs generated, estimated minutes saved, and duplicate reviews avoided.

Clicking a case opens a detail view with: the full affected items table, per-item report reasons and content snippets, a chronological action timeline, and the AI brief — which includes a summary, likely pattern, confidence level, evidence bullets, risk bullets, a suggested modmail draft, and cautions.

Claim locks

A mod claims a case before working it. The claim is recorded with a 60-minute TTL — if the mod goes offline, the claim expires automatically and the case becomes available again. This prevents two mods from simultaneously actioning the same 40 items without any coordination overhead.

Playbook actions

From the detail view, mods choose a playbook action — all gated behind an explicit confirmation step:

• Approve all items in the case
• Remove all items in the case
• Ignore reports on all items
• Lock all posts/comments in the case

After acting, mods mark the case Resolved or Escalated. All actions are logged to the case timeline.

AI Incident Brief

Each new case triggers an asynchronous enrichment job. The GPT-5 model receives anonymized signal data — report counts, reasons, item types, thread structure — and returns a structured JSON object: incident title, narrative summary, likely pattern classification, severity assessment, confidence score, recommended playbook, evidence bullets, risk bullets, a suggested modmail opening, and specific cautions for the moderator. The model's output is constrained to a strict schema; it cannot invent new actions or produce freeform instructions.

Configuration without leaving Reddit

Every subreddit tunes itself differently. SignalDesk exposes a Settings menu action that opens a native Devvit form, pre-filled with the current configuration:

Setting	Default	What it controls
Report Storm threshold	5 reports	Minimum reports on one item to open a Report Storm case
Thread Meltdown threshold	10 reports	Minimum total reports across a thread to open a Thread Meltdown case
AutoMod Flood threshold	8 items	Minimum same-rule filter hits to open an AutoMod Flood case
Window	30 minutes	Time window for all three detectors
AI enrichment	Enabled	Toggle GPT-5 brief generation on/off
Case TTL	30 days	How long resolved cases are retained

Submitting the form writes back through the same configService the detection engine reads from — changes take effect on the next trigger fire.

Reviewer demo seeder

The hackathon review window is short, so we built a Seed Demo Data mod menu action that populates a realistic mod day in one click. It's not a fixture or a hardcoded blob of JSON — it actually exercises the production code path:

• It calls the same service functions the live trigger handlers call (mergeSaveItem, addRecentReportForItem, addRecentReportForThread, addRecentAutomodFilter, upsertReportStormCase, upsertThreadMeltdownCase, upsertAutomodFloodCase). The only thing it skips is the Devvit HTTP routing layer — the Redis schema, deterministic case IDs, severity scoring, and AI job enqueue are all identical to what happens when a real onPostReport trigger fires.
• It dispatches real AI enrichment jobs via scheduler.runJob({ name: 'enrich-case-ai', runAt: new Date() }) — every seeded case actually calls OpenAI and gets a real GPT-5 brief, not a canned response.
• Seven distinct scenarios ship in the seeder, each modeled on a real moderation pattern:

Scenario	Type	Realism
Misinformation Post	Report Storm	Viral health misinformation post, 5 distinct report reasons
Coordinated Brigading	Report Storm	Off-site brigade with rapidly arriving reports from new accounts
Comment Harassment	Report Storm	Single highly-reported comment in a political thread
Political Debate Meltdown	Thread Meltdown	11+ reported comments in one heated thread
Banned Link Domain	AutoMod Flood	Repeated link domain rule firing on multiple posts
Crypto Spam Campaign	AutoMod Flood	Same spam phrase rule hitting 40+ items in 22 minutes
Coordinated Harassment Campaign	URGENT	Cross-thread harassment elevated to CRITICAL severity

A reviewer can click SignalDesk: Seed Demo Data in the mod menu, refresh the dashboard, and see the full incident lifecycle — cases on the kanban, AI briefs streaming in within 3–5 seconds, claim locks, and playbook actions — without waiting for organic events or coordinating multiple test accounts.

Important caveat written into the seeder: because the same code path is used, every seeded case is indistinguishable from a real case in Redis. The dashboard, the scoring, the AI brief — all of it is the production system running on production data. The only artificial part is the source of the signals.

Architecture

Event-driven trigger layer

Every piece of user activity that matters — post reports, comment reports, AutoMod filter hits, mod actions, content deletions — fires a Devvit event handler synchronously. Each handler writes a raw signal record into Devvit Redis and immediately kicks off incident detection for the affected thread. A reconciliation scheduler runs every 5 minutes as a catch-all for events the trigger layer might miss under load.

Incident detection engine

Detection is built around time-window bucketing in Redis sorted sets. Raw signal records are scored by Unix timestamp, so querying for "all reports on post X in the last 30 minutes" is a single ZRANGEBYSCORE call with no table scans. When a threshold is crossed, the engine computes a composite severity score from signal density, report reason distribution, and thread depth, then upserts a case record.

Cases are stored as Redis hashes keyed by case:{caseId}, with a secondary sorted set keyed by severity score to power the kanban board's ordering without scanning all cases.

AI enrichment pipeline

AI enrichment is deliberately decoupled from the trigger path. When a case is created, the case ID is written into a Redis sorted set (ai-jobs) scored by timestamp. A Devvit scheduler job (enrich-case-ai, cron * * * * *) dequeues up to 5 pending jobs per run.

The naive approach — wait for the cron — would mean a 60-second delay. To make briefs feel instant, the detection engine also calls scheduler.runJob() with runAt: new Date(), dispatching the enrichment job immediately in parallel with the queue write. Cases typically have their AI brief within 3–5 seconds of being created.

The OpenAI call uses the Responses API with a strict JSON schema — output is constrained to a typed object with defined fields (title, summary, severity, recommendedPlaybook, evidenceBullets, riskBullets, suggestedModmail, cautions). The recommendedPlaybook field is a string enum drawn from a closed set of predefined playbook strategies. The model has no mechanism to invent new actions or output freeform instructions.

Dashboard proxy architecture

Devvit apps run in a sandboxed environment — there's no public IP, no stable hostname the dashboard can call directly. Instead:

1. The Devvit app generates a short-lived HMAC-signed JWT containing the subreddit ID and moderator identity.
2. The moderator's browser opens signaldesk.arjunganesan.com/?token=<jwt>.
3. Cloudflare Pages validates the HMAC signature, sets a session cookie, and issues a 302 /.
4. All subsequent API calls from the dashboard go to /api/cases/... on the CF Pages worker.
5. The CF Pages worker forwards those requests to the Devvit webview origin with an Authorization: Bearer <devvitContextToken> header.
6. Devvit injects devvit-app, devvit-subreddit, and devvit-user-name headers on the inbound request, which the Hono server uses for all authorization checks.

This means the dashboard never holds Reddit credentials and the Devvit app never exposes a public API — the two sides are connected only through a Devvit-controlled proxy channel.

Claim lock

Case claims use a Redis key with a TTL (case-claim:{caseId} → {modName}, expiry 60 min). If a mod closes their browser mid-investigation, the claim expires automatically and the case becomes available again — no manual intervention, no stuck cases.

The safety constraint we refused to compromise

Every design decision is downstream of one hard rule: the system never takes a destructive action without explicit moderator intent.

• No auto-removes, auto-bans, or auto-locks — ever, under any condition.
• Reporter identity is never stored, displayed, or forwarded to OpenAI.
• AI output is strictly advisory — a label recommendation from a closed set, not a command.
• Every mutating API handler requires a confirmationText field; requests without it are rejected at the handler boundary before any Reddit API call is made.
• Content snippets (title, body, URL) are purged from Redis when the source post or comment is deleted, so the system holds no content that Reddit itself has removed.

Challenges we worked through

• 30-second HTTP fetch ceiling. Devvit's fetch timeout makes synchronous AI enrichment impossible from a trigger handler. The dual-path scheduler dispatch (runAt: new Date() plus the cron queue) is what makes briefs feel instant without ever blocking a trigger.
• No KEYS in Devvit Redis. Every query path has to be served from an explicitly maintained sorted set — there's no scan to fall back on. That forced a clean index design from day one: cases by status, cases by score, cases by type, items by case, reports by item, reports by thread, AutoMod hits by rule.
• OpenAI Responses API output shape varies. The output_text helper isn't always present. The brief parser walks response.output[].content[] defensively and falls back through three shapes before failing — failures are recorded on the case but never break the dashboard.
• Bridging two sandboxes. Devvit can't be called from a public origin, and Cloudflare Pages can't be called from a Devvit trigger. The HMAC-JWT handoff plus session-cookie proxy is the seam that lets a real browser-grade dashboard sit on top of a sandboxed runtime.
• Devvit's per-request async-local-storage. Our first demo seeder tried calling triggersRouter.request() to reuse the HTTP trigger handlers. It silently failed because Devvit binds redis and scheduler per inbound HTTP request — inner requests created from JS don't inherit that context. The fix was to refactor every trigger handler into a thin shell around a pure service function, and have the seeder call the services directly. That made the trigger handlers easier to test, too.

What we're proud of

The enrichment pipeline latency — from case creation to AI brief available in the dashboard — is consistently under 5 seconds in testing, despite running entirely within Devvit's scheduler infrastructure with no persistent server. That required the dual-path dispatch (immediate runJob + queue fallback), strict async/await discipline in the handler to avoid blocking the trigger path, and careful Redis key design so the scheduler can dequeue and lock a job atomically.

Equally proud of what we didn't ship: no auto-removal heuristic, no "trusted reporter" model, no AutoMod rewriter. Every shortcut we cut closed a path to a tool moderators wouldn't trust.

What we learned

• Strict JSON schema mode on the OpenAI Responses API is genuinely production-ready — it eliminated an entire class of "did the model hallucinate a field" bugs and made the AI feel like a typed function call rather than a language model.
• The Devvit scheduler is the right primitive for any external AI workload. Treating triggers as fast normalizers and pushing every LLM call to a scheduled job is the only pattern that actually scales inside the platform's constraints.
• Time-window bucketing in Redis sorted sets is a near-perfect fit for moderation signal aggregation — every detection question collapses to one ZRANGEBYSCORE.

What's next

• Cross-subreddit signal sharing — allow trusted mod networks to share incident patterns (opt-in, anonymized)
• Trend analytics — weekly summaries of incident frequency, resolution time, and playbook usage
• Custom detection rules — let mods define their own pattern thresholds and trigger conditions
• Suggested AutoMod diffs — when an AutoMod Flood case turns out to be a false-positive sweep, generate a YAML change suggestion for mods to review

Built With

• cloudflare-kv
• cloudflare-pages
• devvit-(reddit-platform)
• devvit-redis
• hono
• openai-gpt-5-api
• typescript
• vite