Inspiration
Scientific research is advancing rapidly, especially in AI and Machine Learning, but one fundamental problem remains: reproducibility. A research paper may report impressive accuracy, benchmarks, and experimental results, yet independently verifying those claims is often difficult. Code may be unavailable, datasets may change, experimental configurations may be undocumented, and LLMs can generate convincing but unverified explanations.
We wanted to solve a deeper problem: What if scientific claims could be independently investigated, executed, challenged, and verified through an evidence-first AI system?
That idea led to ReproLab — an Evidence-First AI Research Workbench designed to make computational research more transparent, reproducible, and trustworthy.
What We Built
ReproLab transforms the traditional research verification process into a structured, auditable investigation.
We built a platform combining a 12-persona Constitutional AI Scientist Council with a 14-node LangGraph orchestration pipeline. Each AI scientist has a specialized responsibility, ranging from extracting claims and auditing methodologies to inspecting datasets, evaluating statistical results, and challenging conclusions.
Our system includes:
- Research Paper Intelligence: Ingests academic PDFs, extracts research claims, reported metrics, experimental configurations, and relevant evidence.
- Multi-Agent Investigation: Uses 12 specialized AI scientist personas, each operating under predefined constitutional rules to prevent unsupported conclusions.
- Evidence & Protocol Vault: Uses SHA-256 content-addressed storage, canonical protocol hashing, versioning, and auditable investigation records.
- Secure Execution: Implements AST-based security inspection and isolated execution environments to protect infrastructure from untrusted research code.
- Statistical Verification: Uses Wilson 95% score confidence intervals to compare reported research outcomes against empirical replication results.
- Dual-Lens Interface: Offers a Plain Words view for accessible explanations and a Full Record view for detailed technical evidence, statistical telemetry, and execution logs.
- Bilingual Accessibility: Provides native English and Hindi support to make scientific verification more accessible.
The platform also distinguishes between successful reproduction, partial reproduction, failed reproduction, and technical execution failures rather than treating every successful code execution as scientific validation.
How We Built It
We developed a full-stack architecture using Next.js 16, React 19, TypeScript, and Tailwind CSS for the frontend, with Python, FastAPI, Pydantic, and SQLAlchemy powering the backend.
LangGraph orchestrates the multi-agent investigation workflow, while Google's Gemini models power the AI scientists. Supabase PostgreSQL manages structured research records, and SHA-256 hashing provides content integrity and traceability.
We also integrated OpenTelemetry and Prometheus for observability, alongside a custom statistical visualization called The Rail, which displays claimed and empirically observed confidence intervals.
The frontend is deployed on Cloudflare Workers, with a production-oriented backend and database architecture.
Challenges We Faced
One of our biggest challenges was ensuring that AI-generated conclusions were grounded in actual evidence rather than plausible-sounding language. We addressed this through specialized agent responsibilities, immutable protocol definitions, evidence provenance, and deterministic statistical calculations.
Another major challenge was security. Executing code obtained from academic research introduces serious risks, including arbitrary code execution, unauthorized file access, and network communication. We developed an AST-based security inspection layer and isolated execution safeguards to address these threats.
Coordinating multiple AI agents without infinite loops, unnecessary model calls, or conflicting conclusions was another challenge. We addressed this through bounded LangGraph workflows, specialized agent mandates, and controlled handoffs.
Finally, presenting complex statistical findings without overwhelming researchers required us to rethink the interface. This led to our Dual-Lens experience and native Hindi-English support.
What We Learned
Building ReproLab taught us that trustworthy AI is not simply about using more powerful models. It is about designing systems where models operate within boundaries, evidence is traceable, computation is independently verifiable, and uncertainty is communicated honestly.
We learned to combine agent orchestration, backend engineering, security architecture, statistical reasoning, and human-centered interface design into one cohesive system.
Our vision is to make reproducibility a verifiable engineering process rather than a promise made by researchers. ReproLab aims to become the independent verification layer for computational science — where every claim has evidence, every experiment has an audit trail, and every conclusion can be challenged.
Built With
- css
- genai
- langgraph
- next.js
- opentelemetry
- pydantic
- python
- typescript
