Evidence-Routed Inference: SharpeLab

Inspiration

Large language models are becoming increasingly capable at solving complex problems, yet they often reach different conclusions from the same evidence. In scientific research, finance, medicine, and many other domains, these disagreements frequently arise not because of arithmetic errors, but because different implicit assumptions are being made.

We wanted to investigate whether an AI system could make those assumptions explicit instead of leaving them hidden. Rather than simply generating another answer, our goal was to build a system that identifies which scientific assumptions are actually admissible, routes reasoning through that structured evidence space, and determines whether a conclusion remains valid under every coherent interpretation.

To study this question, we chose the Sharpe ratio as our testbed. Although widely used throughout quantitative finance, the Sharpe ratio appears deceptively simple. In reality, multiple scientifically defensible formulations exist depending on assumptions about return frequency, volatility estimation, annualization, and sampling. This makes it an ideal environment for studying evidence-guided reasoning.

What it does

Evidence-Routed Inference (ERI) is a deterministic reasoning framework that separates evidence from inference.

Instead of asking an LLM to directly produce a final answer, ERI first identifies the relevant assumptions governing a problem. These assumptions define the admissible evidence space. The system then routes reasoning through this structured space, evaluates candidate interpretations, and produces conclusions that are explicitly justified by the selected evidence rather than hidden reasoning paths.

SharpeLab serves as the first scientific demonstration of this approach. Given a financial analysis task, the system identifies the assumptions required for a valid Sharpe ratio specification, evaluates alternative formulations, and determines whether conclusions remain stable across all scientifically admissible interpretations.

How we built it

The project combines deterministic evidence routing with modern LLM reasoning.

We developed a structured evidence ontology for Sharpe ratio analysis, implemented deterministic routing policies that map observations to admissible assumptions, and integrated multiple language models to perform the reasoning tasks within those constraints. Rather than allowing unconstrained agent interactions, each reasoning stage is guided by explicit scientific structure.

The resulting framework lets us compare different reasoning architectures—including monolithic prompting, multi-agent systems, constrained architectures, and deterministic adaptive policies—under identical scientific conditions.

Challenges we ran into

The largest challenge was not engineering—it was scientific design.

Building an evaluation framework required defining what constitutes admissible evidence before any reasoning occurs. Many apparent disagreements between models were ultimately traced back to hidden assumptions rather than reasoning failures.

Another challenge was ensuring that the routing process remained deterministic while still allowing language models enough flexibility to perform meaningful scientific analysis. Designing interfaces that faithfully preserved evidence contracts across different reasoning architectures required substantial iteration.

What we learned

One of our most surprising findings was that additional agents alone do not necessarily improve scientific reasoning.

Our experiments suggest that explicitly structuring the admissible evidence space can have a greater impact than simply increasing the number of reasoning agents. By making assumptions explicit before inference begins, we observed more consistent and scientifically interpretable behavior.

More broadly, this project reinforced the importance of distinguishing evidence selection from inference itself. We believe this separation provides a promising direction for building AI systems whose reasoning is easier to understand, evaluate, and trust.

What's next

SharpeLab is intended as the first demonstration of a broader Evidence-Routed Inference framework.

Future work will extend the approach beyond quantitative finance into scientific research, where many disagreements arise from competing assumptions rather than incorrect calculations. Our long-term goal is to build AI systems that reason transparently by making their evidence pathways explicit before drawing conclusions.

Built With

• codex
• css3
• github
• gpt-5.6
• html5
• javascript
• llm
• multi-agent
• openai
• python