Pulse — Community Intelligence for Game Studios

Inspiration

Every game studio is sitting on a goldmine of signal it can't read. Players don't file neat bug reports — they vent in Discord, swap clips of suspicious opponents, and argue about balance at 2 a.m. Buried in that noise are the three things studios pay real money to learn: who's cheating, what's broken, and how the community actually feels.

We kept seeing the same pattern: a cheat goes viral in a server days before the anti-cheat team notices, or a "minor" bug quietly tanks sentiment for weeks before it shows up in churn numbers. The information existed — it just never reached the people who could act on it. We wanted to build the bridge between what the community knows and what the studio can do.

What it does

Pulse turns raw community chatter into structured, actionable intelligence:

• Cheater signals — surfaces reports of cheating and hacking, each with a confidence score $c \in [0, 1]$ so the team can prioritize.
• Bug intelligence — identifies messages describing bugs and glitches, tagged with severity (low / medium / high / critical) and a description extracted by the LLM.
• Sentiment tracking — classifies every message as positive, negative, or neutral, then aggregates counts per day $N_{\text{neg}}(d)$ into time-series charts so a patch's reception is visible within hours.

Studios get a live dashboard with trend charts and daily AI-generated intelligence summaries — instead of a firehose of raw messages.

How we built it

The pipeline has five stages, two of which use an LLM (Google Gemini):

1. Collect — a Discord bot pulls messages incrementally from authorized channels using checkpoints (only new messages since the last run). Channels are fetched in parallel via asyncio.
2. Classify (LLM) — messages are grouped by channel and sent in batches of $B = 20$ to the Gemini API. A single structured prompt classifies all four categories at once. The LLM is constrained to a Pydantic schema, returning per-message flags and confidence scores as valid JSON — no text parsing needed.
3. Aggregate — a SQL query in BigQuery rolls up per-category counts into daily totals:

$$N_{\text{category}}(d) = \sum_{m \in \text{messages}(d)} \mathbf{1}[\text{flag}_{\text{category}}(m) = \text{true}]$$

1. Summarize (LLM) — a second LLM pass reads the day's classified messages and generates three concise summaries (sentiment, cheaters, bugs) plus per-channel and server-wide narrative intelligence.
2. Present — a FastAPI service reads from BigQuery and renders a dashboard with time-series line charts (selectable range: 7–180 days), collapsible panels, and detailed AI summaries selectable by date.

All data is stored in Google BigQuery across four tables: raw messages, per-message analysis results, daily aggregates, and daily LLM-generated summaries. The tech stack is Python, discord.py, Google Gemini API, BigQuery, FastAPI, and Chart.js.

Challenges we ran into

• Signal vs. noise. Discord is sarcasm, memes, and inside jokes. "this game is so broken lol" can be praise or a real bug report. Getting the classifier to read tone and only flag genuine issues took the most prompt iteration.
• Structured output reliability. Getting the LLM to consistently return valid JSON matching our Pydantic schema for every message in a batch — including edge cases like very short messages ("gg") or mixed-language text — required careful prompt engineering and schema design.
• Two-stage LLM design. We started with classification only, then realized numbers alone don't tell the story — a studio lead needs to read "what happened today" in 30 seconds. Adding summarization as a separate second stage with its own prompts let us tune each stage independently.
• Confidence over false certainty. An accusation is not a verdict. We deliberately ship ranked confidence, never automated punishment — the studio's team always makes the call.

What we learned

• Communities detect problems before metrics do; the bottleneck is structure, not data.
• One LLM call with all four categories is faster and cheaper than four separate calls, with no measurable accuracy loss — batching matters.
• Prompts stored in Markdown files (not hardcoded) made iteration dramatically faster — we could tune classification behavior without touching application code.
• The simplest pipeline that works end-to-end is more valuable than a sophisticated one that doesn't ship. We started with classify-only, validated it worked, and only then added summarization.

What's next

• Deploy to Google Cloud Run with scheduled daily pipeline runs.
• Per-game tunable categories and risk weighting so each studio can calibrate what "risk" means for their title.
• A feedback loop where studio confirmations improve classifier prompts over time.
• Expansion beyond Discord to other community surfaces, using the same consent-first collection model.

Built With

• anthropic
• docker
• fastapi
• github
• next.js
• postgresql
• python
• react
• redis
• tailwind
• typescript