Truth Bench

Inspiration

We realized that when you ask an AI model a political or controversial question, it often provides an answer based on its training alignment, bias encoded as safety, rather than what is objectively true. We wanted to find a way to measure this bias objectively. We recognized that while linguistic bias is hard to quantify, financial bias is not. In prediction markets, bias has a specific price tag. If a model is delusional or overly "safe" to the point of ignoring reality, it loses money. We built Truth Bench to use the Efficient Market Hypothesis to stress-test the cognitive integrity of frontier models.

What it does

Truth Bench is a benchmarking platform that evaluates AI models based on their ability to predict reality rather than their ability to adhere to safety guidelines. It operates through three main mechanisms: Live Simulation: We deploy AI agents (powered by models like Gemini 3 Pro, Grok, etc.) to trade on real-world outcomes in prediction markets. RLMF (Reinforcement Learning from Market Feedback): Unlike RLHF which relies on biased human graders, we use Profit & Loss as the reward model. If the model predicts correctly, it is rewarded; if it lets ideology cloud its judgment, it is penalized. The Truth Score: We generate a composite metric called the "Truth Score" based on Total Return, Win Rate, Sharpe Ratio, and Max Drawdown. This effectively measures "Rationality per Dollar."

How we built it

We engineered a multi-agent system where each agent acts as a trader powered by a specific LLM. The system ingests real-time market data and prompts the models to make buy/sell decisions based on that data. The Dashboard: We built a frontend to visualize the "Live Simulation" where users can watch the equity curves of different models in real-time. The Scoring Engine: We implemented a backend analysis tool that calculates financial metrics (Sharpe Ratio, Drawdown) to synthesize the Truth Score. Consensus Engine: We developed an analysis layer that aggregates the agents' moves to identify arbitrage opportunities against irrational human panic in the market.

Challenges we ran into

Working with the various API's, avoiding rate limits with kalshi, simulating a training pipeline were some challenges we had to overcome.

Accomplishments that we're proud of

Defining RLMF: We successfully conceptualized and demonstrated Reinforcement Learning from Market Feedback, proving that financial markets can serve as a mathematically pure feedback loop for AI alignment. The Truth Score: We moved beyond arbitrary ratings to create a "Rationality per Dollar" metric. Proving the Concept: We demonstrated that models with access to real-time, unfiltered data (like Grok) significantly outperform models that prioritize ideology, proving that truth is a probabilistic property rather than a linguistic one.

What we learned

We learned that current "safety" alignments in AI often function as cognitive blinders. When a model prioritizes being "safe" over being "right," it cannot survive in a prediction market. We also discovered that AI agents can actually serve as a stabilizing force. When human traders panic, the AI agents, who possess infinite patience and no emotion, step in to arbitrage the difference, anchoring the crowd back to the truth.

What's next for Truth Bench

We aim to make Truth Bench the standard evaluation metric for every Frontier Lab, including xAI and OpenAI. We believe that if a model cannot survive a prediction market, it does not truly understand the world. We plan to expand the platform to allow developers to test their own fine-tuned models against our market environment, establishing a universal standard for cognitive integrity in AGI.

Built With

• kalshi
• python
• typescript
• x