Inspiration
When Jensen Huang resigned from Nvidia, the Washington Post reported that "online bots" were rushing to defend him. Claims like that shape how a story is covered, and almost nobody can check them. When thousands of accounts repeat something, it reads as consensus, and AI has made that volume nearly free to produce.
We wanted a tool that answers one question in seconds, from public data: how much of the noise behind a claim is real people? Our first idea was a simulator for running narrative campaigns. We dropped it because it would help the people we want to catch. Headcount only measures.
What it does
Text Headcount a claim, or type one into the web app. It finds every Bluesky post making that claim, sorts them into the narratives people are telling, scores every account behind them for automation, and compares the result to the whole story and to what wire services and public broadcasters reported.
- iMessage agent: a two-line answer, a one-image summary card, and a numbered menu for follow-ups (narratives, what the automated accounts share, who spreads them, newsroom coverage).
- Any story, live: for claims outside our corpus it searches Bluesky in real time, clusters narratives on the fly and scores the accounts, in 13 to 26 seconds.
- Newsroom dashboard: a guided six-chapter investigation of the Huang week.
- Spot the bot: a game where you try to beat the model at picking out automated accounts.
It never rules on whether a claim is true, never says who runs an account, and never names private accounts.
How we built it
Data. From Bluesky's open AT Protocol we pulled 46,656 posts about the story from 20,495 accounts, over 1 million posts from those accounts' histories, and 380,164 likes, reposts and quotes. We also collected 340 headlines from ten newsrooms, including Reuters, AP, AFP, the BBC, NPR and Al Jazeera.
Content layer. DeepSeek labeled all 40,197 unique post texts into narratives and sides. We trained a lightweight embedding classifier on those labels so live checks don't need a language model call per post. Claim matching uses hybrid retrieval (bge-small embeddings plus TF-IDF), which raised recall@20 from 0.575 to 0.787.
Connectivity layer. For each account we compute 39 behavioral signals (posting rhythm, volume, repetition, how others engage with it) plus 3 coordination signals. Accounts that share the same link, repost the same post within 60 seconds, or post near-identical text are linked into coordination clusters: 249 clusters covering 574 accounts.
The bot model. There's no labeled set of bots, only 287 accounts that admit to being automated. We used positive-unlabeled learning: train a classifier to find the self-labeled bots from behavior alone (never the label or the bio), then rescale its score into an estimate of automation (Elkan and Noto, 2008):
$$P(\text{automated} \mid x) = \frac{P(\text{labeled} \mid x)}{c}$$
where c is the model's mean score on held-out labeled bots. We evaluate by hiding labeled bots and checking how many land in the top k:
$$\text{recall@}k = \frac{|\,\text{top-}k \cap \text{held-out bots}\,|}{|\,\text{held-out bots}\,|}$$
Recall@100 is 0.356 ± 0.016 across five splits, about 8 times the 0.045 you get by ranking on posting rate. Precision@50 is 0.96, it catches 23% of bots from operators it never saw in training, and a manual review of 50 top-scored unlabeled accounts found 46 automated.
Honest uncertainty. One account's posts aren't independent, so every share gets a 95% interval from a bootstrap that resamples accounts, 1,000 times:
$$\hat{s}^{(b)} = \frac{\sum_{i \in b} y_i}{\sum_{i \in b} n_i}$$
where account i has n_i posts and y_i of them come from a likely automated account.
Serving. A FastAPI service runs the analysis, writes every reply from templates in code, and renders the image card. A LangChain deepagent on DeepSeek runs the conversation, with LangSmith tracing. Photon connects it to iMessage. The React app runs on Vercel at demoyk.com.
What we found
- No bot campaign defending Huang on Bluesky. Posts defending him were 4.8% likely automated, the same as the story's 4.5% baseline. The pattern the Post reported doesn't show up here.
- Jokes and memes were almost entirely human, at 0.4% automated. What automation there was leaned toward accounts relaying the news.
- Caution disappears in the retelling. 54% of newsroom headlines on the spy allegation hedged ("alleged", "suspected"). Bluesky posts sharing a link kept the hedge 50% of the time. Posts written in people's own words kept it only 29% of the time.
- No "gullible type" of amplifier. Across 630 real accounts that amplified automated posts, none of our behavioral archetypes was more likely to do it than matched accounts.
Challenges we ran into
- No ground truth. Nobody publishes a list of bots, so we built the model around the few accounts that admit it and validated it four ways: held-out bots, unseen operators, a posting-rate baseline and a manual review.
- Posts aren't independent. Prolific accounts would have made naive intervals far too narrow, so we resample at the account level.
- Safety rules can't live in a prompt. In rehearsal the language model declined our flagship claim as a question about a private person. We moved every safety rule into code: the API decides what to decline, and templates write every number.
- Privacy at scale. Some Bluesky users opt out of logged-out viewing through a label clients have to honor. We made it override everything and swept every export independently: zero opted-out accounts exposed.
- Two AI agents, one codebase. Our coding agents built in parallel through a shared plan, a message log and pull requests, each owning its own folders, with continuous integration on every merge.
Accomplishments we're proud of
- A working product on three surfaces (iMessage, web, dashboard) that checks any story live, not just our demo.
- A bot model 8 times better than the obvious baseline, trained without a single hand-labeled bot.
- Every number on screen traces to an evaluation report in the repo, with intervals.
What we learned
Measuring is harder than generating. The interesting work was proving what the data does and doesn't show, then building a product that says exactly that and nothing more.
What's next
Hand-labeled validation sets for narratives and bots, coordination detection aimed at campaign-style networks, more platforms, and a dedicated iMessage line so anyone can text Headcount without signing up.
Built With
- bluesky-at-protocol
- claude
- cloudflare-tunnel
- deepseek
- elevenlabs
- fastapi
- github-actions
- langchain-deepagents
- langgraph
- langsmith
- matplotlib
- numpy
- photon-2.0-(photonhq-api)
- playwright
- polars
- pyarrow
- pytest
- python
- react
- scikit-learn
- sentence-transformers-(bge-small-en-v1.5)
- typescript
- umap
- vercel
- vite
