Inspiration
AI systems can produce confident forecasts and recommendations, but confidence is not the same as evidence. I wanted to build a system that treats an AI model like a scientific hypothesis: compare it against a baseline, test it against real outcomes, preserve the evidence, and require human review before deployment.
What it does
The Scientific Method for AI is an evidence-first evaluation system for AI-generated forecasts and recommendations.
It compares a candidate model against a baseline, measures performance using real results, preserves an audit trail, checks for data-integrity problems, produces a clear recommendation, and keeps the final decision with a human reviewer.
The demo shows the same evaluation framework applied to both delivery forecasting and soccer forecasting.
How I built it
I built the project using Python, structured data pipelines, automated tests, reporting tools, and a dashboard that presents the results clearly.
Codex helped inspect the codebase, implement and verify changes, run tests, diagnose workflow issues, improve the dashboard, and prepare the final demo. I directed the product vision, evaluation philosophy, and architecture.
Challenges I faced
The hardest challenge was preventing the system from becoming another black-box predictor.
I solved that by separating prediction from evaluation. Models generate predictions, while the research engine independently grades, analyzes, and reports the results.
Other challenges included preventing silent data changes, preserving raw source data, rejecting invalid inputs, keeping predictions traceable to model and code versions, and making technical evidence understandable to a human reviewer.
What I learned
I learned that reliable AI development is not only about producing better outputs. It is about creating a trustworthy process for proving whether those outputs are actually better.
I also learned that Codex is most effective when it works inside a disciplined process with explicit goals, automated tests, verification, and clear stopping points.
What’s next
The next step is to expand the shared research engine so more models and domains can plug into the same evaluation process.
Future work includes broader automated experimentation, model-version comparisons, stronger anomaly detection, richer reporting, deployment approval workflows, and a reusable evaluation platform for teams building AI systems.
The long-term goal is to make evidence-based AI evaluation as normal as automated software testing.
Log in or sign up for Devpost to join the conversation.