verity by Ashmoliens

Inspiration

Finding patterns in data is easy, knowing which one to trust is hard. Try 20 trading rules and one may look amazing by chance, if you try 200 and the winner looks even better. A LLM makes that even worse as it could propose a fresh batch of rules every time you ask.

What it does

You give verity a thesis, such as "an inverted yield curve predicts equity drawdowns." A generator turns that sentence into about ~20 rules which consist of: feature, series, lookback, transform, asset and direction which are passed into a statistical deterministic engine.

This statistical engine then goes through three checks of all points between 2003-2014. 1) Editing the future - scale everything by 5% and recompute. 2) Significance - ensure that stationary bootstrap p-value has to be clear 0.05 3) Luck Hurdle - shift the rules out of time and re-run the whole search 200 times.

After each round, the result is passed back into the AI reviewer to do one of the three things. 1) Challenge a rule with an additional test - resulting in a pass or fail. 2) Refine - writing at most five new rules that re-enter the same ledger. 3) Nominate at most three names then stop.

Finally, the code assigns one of five labels: Rejected, Can't tell yet, Fragile, Promising, Replicated.

How we built it

We built it in Python, utilising libraries like Pydantic, Pandas, Numpy. For our demonstration we have used Streamlit to provide a quick and easy interface to test our underlying logic.

Challenges we ran into

The engineering aspect of it was extremely open. The decisions we have taken with the AI reviewer, the deterministic engine were our interpretation of the problem statement.

The hardest design problem was keeping the model out of the final decision. Obviously, LLMs tend to modify the rules to ensure a "correct" output. A reviewer that is able to see the validation or holdout results would definitely produce incorrect results.

What we learned

Multiple testing is a property of the search and not of the winning rule. Power belongs on the verdict. An insignificant result in a short sample is withheld belief. Saying, "Can't tell yet" is a different claim from saying "We looked and did not find anything there" and the minimum detectable Sharpe ratio is what makes the difference computable.

A LLM is a good drafter of hypotheses. The useful constraint was a narrow interface: only six fields and a variety of tests which are designed to hurt a rule.