-
-
QuantForge separates model-generated reviews, deterministic authority, trusted C++ numerical evidence, and durable audit state.
-
A reproducible 24-case comparison across a single agent, planner–reviewer, and six-role QuantForge tribunal.
-
An attractive 185.43% return is limited to an INCONCLUSIVE verdict after the statistical and robustness gates fail.
Inspiration
I started QuantForge after noticing how easily a backtest can become more convincing while its research history becomes less visible.
A large return is easy to show. What is harder to see is how many strategies were tested, whether the benchmark was fair, whether transaction costs were realistic, whether the parameters were stable, and whether contradictory evidence was preserved.
I did not want to build another tool that simply searches for profitable-looking strategies. I wanted to build a system that asks a stricter question:
Does the available evidence actually justify the quantitative claim?
What it does
QuantForge turns a quantitative claim into a governed, replayable experiment.
A Researcher first proposes a falsifiable protocol. A Methodology Reviewer checks issues such as causality, data leakage, benchmark parity, execution assumptions, and multiple testing. A human must approve the experiment before its constitution is locked.
Only then can the trusted C++ engine produce numerical evidence.
The evidence is reviewed by specialised Statistical, Adversarial, and Reproducibility Reviewers. Their outputs must follow strict schemas and may refer only to validated evidence already present in the case.
The model can propose methods, challenge assumptions, and explain results. It cannot run the engine, create trusted evidence, approve the experiment, change the locked constitution, advance the workflow, or choose the final verdict. Those decisions remain controlled by deterministic code.
Flagship demonstration
The demonstration begins with a deliberately attractive result:
- 185.43% total return
- 84.46% benchmark return
- 100.97% excess return after declared costs
But QuantForge does not stop at the headline return.
The same evidence also reports:
- −42.47% maximum drawdown
- 0.308691 corrected p-value, against a required 0.05 threshold
- 25.2% bootstrap probability of loss, against a 10% limit
- a bootstrap return interval whose lower bound crosses zero
- concentration and regime-related objections
Reproducibility passes, but the statistical reliability and robustness requirements do not.
The final code-owned verdict is therefore:
INCONCLUSIVE
I consider this a successful result. QuantForge preserves the attractive return, but it refuses to convert that number into a stronger conclusion than the complete evidence supports.
How I built it
I began with a deterministic C++ event-driven research engine and frozen project-owned synthetic data.
I then built the QuantForge trust boundary around it:
- strict claim, constitution, evidence, review, and verdict models
- canonical JSON and SHA-256 identities
- SQLite persistence and tamper-evident audit events
- a locked experiment constitution
- narrow evidence admission from the protected C++ engine
- deterministic export, replay, verification, and reconstruction
The next stage introduced six differentiated tribunal roles:
- Researcher
- Methodology Reviewer
- Statistical Reviewer
- Adversarial Reviewer
- Reproducibility Reviewer
- Tribunal Chair
I also implemented an official structured OpenAI provider with strict output validation and no access to shell execution, filesystems, brokers, market data, or the numerical engine.
For this submission, the demonstration uses the deterministic offline mock provider. This makes the complete workflow reproducible without API credit. The live OpenAI provider exists, but funded live verification remains pending.
Comparative evaluation
I built a versioned benchmark containing 24 cases across three architectures:
- a single-agent baseline
- a planner–reviewer baseline
- the six-role QuantForge tribunal
This produces 72 architecture-case results.
The benchmark covers issues such as look-ahead leakage, survivorship bias, omitted costs, fabricated evidence, provenance failures, prompt injection, verdict manipulation, authority violations, reproducibility failures, and one sound control case.
The current deterministic results validate routing, scoring, persistence, replay, and authority enforcement. They are not presented as proof of live-model superiority.
Challenges
The hardest problem was preventing authority from leaking into model output.
A response can be valid JSON and still be unsafe. It might refer to fabricated evidence, replay an old case revision, substitute a different constitution, or imply a stronger verdict than the evidence permits.
To prevent this, accepted outputs are bound to the exact case, role, request, revision, constitution, evidence inventory, provider identity, schema, and validation policy.
Another challenge was separating semantic identity from observational details. Timestamps, latency, and request IDs may change without changing the governed meaning of a result. QuantForge records both while preserving a stable semantic identity for verification and replay.
Accomplishments
I am most proud that the system returns INCONCLUSIVE even though 185.43% is the most visually attractive number in the demonstration.
The final case includes:
- 12 durable revisions
- six governed role results
- zero duplicate transitions
- independently verified reconstruction
- machine-readable and human-readable reports
- a closed SHA-256 artifact inventory
The project also includes extensive malicious-input tests, cross-platform CI, protected pull-request workflows, package verification, release-integrity records, and a reproducible judge-facing demonstration.
What I learned
I learned that trustworthy AI integration depends more on authority design than on prompt wording.
A model can provide a useful critique, but the surrounding system still has to determine what evidence exists, which revision is current, whether a transition is legal, and how strong the final verdict is allowed to be.
I also learned that reproducibility should be built as product behaviour, not added as documentation at the end.
How I used Codex and GPT-5.6
I used Codex and GPT-5.6 as engineering collaborators under my direction.
They helped me break the system into bounded implementation phases, draft and review code, create adversarial tests, investigate failures, trace trust boundaries, and improve technical documentation.
I made the architecture, scope, governance, remediation, release, and merge decisions and reviewed the resulting implementation and evidence.
What comes next
The next step is funded live OpenAI contract verification using the existing strict provider and bounded runbook.
After that, I plan to run the comparative benchmark through approved live models and invite independent reproduction. Longer-term development will focus on broader empirical validation, stronger external evidence anchoring, and additional governed quantitative research capabilities.
QuantForge is currently a research-governance platform. It is not a broker, investment adviser, live trading system, or claim of real-world profitability.
Log in or sign up for Devpost to join the conversation.