SkepticR

Inspiration

AI can test thousands of ideas quickly, but some may look successful just because of luck. We wanted to build an AI that checks whether its discoveries are real before reporting them.

What it does

SkepticR is an AI that checks whether its ideas are genuinely good or just lucky.

AI Researchers generate ideas about stocks, health claims and experiments.
A Ledger records every idea and decision.
SkepticR tests ideas in hundreds of fake situations where nothing is real to see if they could succeed by chance.
Skeptics check for problems such as using future data, ignoring failed companies, trading costs and misleading comparisons.
A Judge gives each idea a final result: Looks real, Probably luck or Not enough data.
The final report only uses ideas that pass the checks.

How we built it

Backend: Python, FastAPI, NumPy and SciPy for data processing and statistical testing.
Frontend: React, TypeScript and Tailwind for the user interface.
AI: Claude or Gemini can generate ideas and explanations, but they do not perform the statistical tests.
Research: Searches Crossref, arXiv and OpenAlex for real research papers.
Offline mode: Works without an API key using built-in mock AI and recorded demos.

Challenges we ran into

Making fake data realistic enough to test luck properly.
Preventing the credibility score from reaching zero and blocking future discoveries.
Preventing similar ideas from repeatedly earning credibility.
Avoiding fake research papers and links generated by AI.
Handling stock data that ignores companies that failed.

Accomplishments we're proud of

We tested SkepticR using 500 signals, including 5 real ones, across 20 runs.

Naive AI had 90% false discoveries.
Traditional statistical methods had 64–71% false discoveries.
SkepticR had 0 false discoveries out of around 80 discoveries, while still finding around 80% of real signals.

It also detected misleading health data and a fake betting advantage caused by random results.