Inspiration
We live in a world where predictions shape reality. Election odds move markets, social media hype drives behavior, and AI opinions influence decisions—yet no one consistently asks who is actually right. We were inspired by this gap between confidence and correctness and set out to build the analysis we wished existed: a system that measures prediction credibility, not just prediction popularity.
What it does
Signal vs Noise is an AI-powered Prediction Credibility Engine that evaluates how accurately different sources—prediction markets, social signals, and AI-generated forecasts—anticipate real-world outcomes. The project aligns predictions with ground-truth results and scores each source on accuracy, calibration, and early-signal strength, revealing which signals deserve trust and which are pure hype.
How we built it
We used Hex as a unified analytics workspace to ingest public datasets from prediction markets, social platforms, and verified outcome sources. We aligned all signals on a shared timeline and evaluated them using probabilistic scoring methods such as the Brier Score:
Brier Score
1 𝑁 ∑
𝑖
1 𝑁 ( 𝑝 𝑖 − 𝑜 𝑖 ) 2 Brier Score= N 1
i=1 ∑ N
(p i
−o i
) 2
We then built interactive dashboards and data apps to compare sources, visualize confidence vs. accuracy, and surface credibility rankings. Hex’s AI features were used to generate hypotheses, explain anomalies, and enable conversational exploration of results.
Challenges we ran into
Aligning predictions before outcomes without introducing hindsight bias
Normalizing confidence signals across very different data sources
Designing metrics that reward calibration, not just boldness
Turning complex statistical results into intuitive visual stories
Accomplishments that we're proud of
Built a unified framework to score prediction credibility over time
Created a dynamic leaderboard ranking markets, crowds, and AI models
Identified “hype zones” where confidence was high but accuracy collapsed
Delivered an interactive, shareable data app that feels native to Hex
What we learned
We learned that the most confident signals are often not the most accurate—and that calibration beats conviction. Data storytelling is just as important as methodology; insights only matter if people can understand and trust them. We also learned that AI becomes most powerful when it helps explain why something failed, not just what happened.
What’s next
Next, we plan to expand the engine across more domains, introduce real-time credibility scoring, and add causal analysis to explain why certain signals outperform others. Long term, this system could serve as a trust layer for decision-making, helping people navigate a world increasingly driven by predictions.
Log in or sign up for Devpost to join the conversation.