Paper Trail — Devpost Submission

Inspiration

I've recently been quite interested into the propogation of dedicated messaging through social media. People, especially those who lack technical knowledge, are particularly vulnerable to these social media campaigns. It is particularly scary, though, how many people are affected by even long term subtle messaging. This is particularly bad with smear campaigns and coordinated inauthentic behaviour, that often rely on the same message being copied or paraphrased across many accounts. Paper Trail was my attempt trace a single narrative back to plausible origins and to surface when the same message is being pushed by multiple accounts, hopefully giving users and researchers a clear, evidence-backed view of how a claim propagated.

What it does

Paper Trail is a narrative provenance tracker with a Chrome extension for Bluesky. One click on a post sends its text to our backend, which:

• Builds a corpus by searching Reddit (NOT FULLY IMPLEMENTED CUZ I DONT HAVE API YET) and Bluesky (AT Protocol) in parallel using a query expansion: a post snippet, n-gram phrases, and LLM-extracted search phrases (capped at MAX_SEARCH_QUERIES), with a 7-day corpus window.
• Clusters and orders posts by embedding similarity and timestamp, then builds a provenance graph (DAG): nodes are posts (corpus + current), edges are supported by multi-signal evidence (quote overlap, n-gram match, paraphrase score). The graph is ordered by time and can branch (one node -> many).
• Tracks one message: I extract a core claim from the current post and score each node for whether it carries that message (verbatim / paraphrased / shifted). I (try to) detect when the same message appears across many accounts and flag it in the report.
• Runs structural rules (deterministic: timestamp order, origin-in-corpus) and semantic verification (embeddings + optional LLM) so that mutations and diff phrases are tied to corpus evidence; agents abstain when evidence is weak.
• Synthesises a user-facing summary: one-liner, confidence (low/medium/high), origin snippet, and an optional full report with the provenance graph (nodes labelled by timestamp, coloured by confidence), propagation stats, evidence spans, and a ready-to-copy reply draft.

The extension injects a Trace button on bsky.app; the result is shown in an inline panel (origin, timeline, diff, propagation block, copy reply).

How we built it

• Backend: FastAPI (Python 3.10+), single entrypoint (/trace). Orchestrator in main.py runs scrapers in parallel, then agents sequentially: clustering → provenance graph → narrative diff → structural rules → semantic verifier → synthesis. All config (timeouts, thresholds, MIN_SIGNALS_FOR_EDGE, CORPUS_DAYS_LIMIT, etc.) lives in config.py; prompts in prompts.py.
• Scrapers: scrapers/reddit.py (PRAW, time_filter="week", up to MAX_SEARCH_QUERIES per run), scrapers/bluesky.py (atproto searchPosts). Both consume the same keyword list from _extract_keywords() (snippet + n-grams + LLM).
• Agents: agents/provenance_graph.py builds the DAG and computes propagation; agents/edge_evidence.py scores edges (quote/ngram/paraphrase); agents/message_propagation.py extracts the message and scores nodes; agents/diff.py, agents/reply.py, agents/structural_rules.py, agents/semantic_verifier.py, agents/synthesis.py. Optional connection-sanity LLM gate; we keep it off by default and rely on multi-signal edges.
• Reliability: Multi-signal edges (≥2 of quote/ngram/paraphrase), abstention when evidence is weak, origin-in-corpus rule, confidence from rules + mutations + unverified cap, evidence spans in the report, temperature=0 for LLM calls, fallbacks for message and keyword extraction.
• Extension: Chrome extension (manifest v3) with content script on bsky.app, CORS to backend; panel shows UserSummary and optional full report HTML from report.generate_report().
• Audit: Append-only mutation log (e.g. JSONL) for trace ID, agent ID, mutation type, spans, confidence.

Challenges we ran into

Ooooh boy. There were a LOT.

• Rate Limits and API Problems: Yeah so social medias don't like it when you try your best to run through them and their users. I kept getting ratelimited in most social medias except Bluesky, which is why I stuck with it.
• Overfitting to one lineage: My early designs assumed a single chain (A→B→C). But I found real propagation is a network: one post leads to many. I tried relaxed config (MIN_EDGES_TO_USE_GRAPH, MIN_SIGNALS_FOR_EDGE) and support branching but I don't think it was fully implemented correctly.
• Hallucination: LLM mutation notes and diff phrases had to be grounded. I added structural rules (deterministic), semantic verification (embeddings/LLM vs corpus), evidence spans in the report, and confidence capping when claims are unverified.
• Same event vs similar: We tightened edge criteria so “similar” means same story: multi-signal evidence, relevance thresholds (THETA_RELEVANCE, THETA_EDGE), and optional connection-sanity check so edges reflect real derivation, not just topical overlap.

Accomplishments that we're proud of

• One-click provenance from a Bluesky post to a time-ordered, evidence-backed graph and propagation view, with a clear confidence badge and optional full report.
• Multi-signal, evidence-bound edges: no edge without quote/ngram/paraphrase support; agents abstain when they can’t cite the corpus.
• Message propagation and same-message detection: tracked message, verbatim/paraphrased/shifted breakdown, and an explicit “Same message across N accounts” when many authors share the same text—useful for smear/coordinated-campaign awareness.
• Determinism and audit trail: temperature=0, stable sorting, append-only mutation log, and a single config/prompt surface so behaviour is reproducible and tunable.
• Robust search pipeline: snippet + n-grams + LLM keywords, shared across Bluesky with a 7-day window, so the corpus is more likely to contain the real origin and siblings.

What we learned

• Provenance is inherently multi-hop and branching; a single linear timeline is too restrictive. Modelling it as a DAG with a “main path” plus alternative paths and propagation nodes works better.
• Search is the bottleneck: if the corpus doesn’t contain the origin or related posts, downstream agents can’t recover. Investing in query expansion (snippet, n-grams, LLM) and using the full query budget in both scrapers paid off.
• User-facing output should stay minimal (one-liner + confidence); the full graph, evidence spans, and propagation stats are there for power users and research.

What's next for Paper Trail

• More platforms: add Twitter/X or other scrapers behind the same keyword and schema interface.
• BENCHMARKING, BENCHMARKING, BENCHMARKING: No seriously, how do I benchmark this?
• Internal dashboard: expose the full provenance graph, mutation graph, and clustering (e.g. dendrogram or 2D embedding projection) for researchers.
• Tuning and thresholds: use the mutation log to analyse confidence vs. human judgement and tune THETA_*, MIN_SIGNALS_FOR_EDGE, and propagation thresholds.
• Caching and rate limits: cache embedding and search results per post fingerprint to reduce API calls and respect platform rate limits.

Built With

• a-lot-of-luck
• atproto
• chrome-manifest
• python