Inspiration
For a first-time NLP learner, “reproduce this paper” is rarely a single task. The learner must understand the paper’s claim, find the relevant repository, separate the paper dataset from local teaching data, prepare an environment, record a run, compare unlike results honestly, and explain what is still missing. For learners without a computer science background, those decisions are easy to lose inside commands, browser tabs, and partial notes.
ReproPath was created to make that process explicit. Instead of treating reproduction as a yes-or-no claim, it gives the learner a fixed path, requires evidence at each stage, and preserves the boundary between a method-level smoke test and a paper benchmark.
What it does
ReproPath guides a learner through one curated fastText reproduction case using a fixed seven-checkpoint protocol:
- Understand the task
- Connect the repository
- Confirm data and metric
- Prepare the environment
- Run the minimal target
- Compare results
- Record gaps
The first four checkpoints present reviewed, read-only evidence with source URLs or repository locators. Checkpoint 5 holds an editable run record. Checkpoint 6 derives the comparison from the recorded paper and local evidence. Checkpoint 7 preserves unresolved gaps and allows learner notes and additional gaps.
Status is evidence-gated and deterministic: removing a required checkpoint-5 item changes that checkpoint to in progress, blocks later checkpoints, and disables continuation. Learners cannot simply select a successful outcome.
The curated case uses the paper Bag of Tricks for Efficient Text Classification and the official fastText repository. Its bundled mini-news files are project-authored teaching fixtures, not AG News. The observed local P@1 of 0.875 supports only a bounded method-level smoke test. ReproPath keeps the paper benchmark marked as not comparable and does not claim to reproduce it.
The completed record becomes a Reproduction Passport v2. The on-screen preview and downloadable Markdown and JSON are generated from the same validated Passport object, including all seven checkpoints, provenance, comparison scope, gaps, missing evidence, and claim boundaries. Project state is stored in the learner’s browser, and a confirmed reset restores the reviewed demo seed.
How we built it
We built ReproPath as a client-side React and TypeScript application with Vite. React Router provides direct routes for Intake, the Reproduction Map, each checkpoint, and Passport. Zod validates project and Passport schemas, while deterministic domain rules derive checkpoint availability, checkpoint status, project status, comparison scope, provenance, and missing evidence.
The current release stores one curated project in browser localStorage. A narrow hydration adapter opens the earlier milestone’s stored project as a schema-version-2 project without turning the code into a general migration framework. Passport Markdown and JSON are serialized from one validated version-2 object.
The repository includes focused domain, storage, hydration, component, reset, Passport, and failure-path tests. The final release verification passed 11 test files and 78 tests. GitHub Actions runs type checking, linting, tests, and a production build on pull requests and pushes to main. The production application is deployed on Vercel.
How Codex and GPT-5.6 were used
GPT-5.6 Thinking supported product scoping, research-boundary review, milestone critique, evidence-contract review, and submission planning.
Codex implemented, tested, reviewed, documented, and deployed the React application through the primary build task. Human review controlled each stage boundary and kept sourced facts, observed run evidence, and unsupported claims distinct.
GPT-5.6 Thinking was part of the development and review workflow. The production application does not call GPT-5.6 at runtime and has no model or API integration.
Challenges
The hardest challenge was making the claim boundary visible without making the learning flow feel defensive. The local fixture is intentionally small and useful for a method smoke test, but it is not AG News and its P@1 is not numerically comparable with the paper’s AG test accuracy. That distinction had to remain visible in the map, checkpoint derivation, gap record, and Passport.
A second challenge was ensuring that status followed evidence rather than user preference. Ordered prerequisites, incomplete evidence, failed runs, and derived comparisons all needed one consistent rule set across navigation, page summaries, persistence, and exports.
The third challenge was recovery and provenance. The application needed to preserve valid browser state, open the earlier stored project safely, distinguish untouched curated evidence from learner edits, and reset only the ReproPath project without clearing unrelated browser storage.
Accomplishments
- Delivered a complete seven-checkpoint learning workflow at a working public URL.
- Made checkpoint and project status deterministic and evidence-gated.
- Attached source URLs or repository locators to curated paper and repository claims.
- Preserved an explicit distinction between the method smoke test and the unreproduced paper benchmark.
- Built Passport v2 with one shared object for the preview and Markdown/JSON exports.
- Added focused coverage across domain rules, storage, hydration, UI behavior, reset, exports, and failure paths, reaching 11 test files and 78 passing tests in the final release verification.
- Added GitHub Actions verification and a production Vercel deployment.
What we learned
Reproduction becomes more teachable when it is framed as a sequence of evidence decisions rather than a final score. A small, sourced case can teach more than a broad automated workflow if it makes the target, prerequisites, provenance, and unsupported claims visible.
We also learned that a “successful run” and a “reproduced benchmark” are different statements. Encoding that distinction in the data model and the interface is stronger than relying on a warning at the end. Finally, deriving every status and both exports from shared validated state reduces contradictions and makes the learner’s record easier to inspect.
What is next
The current release intentionally supports one curated fastText case. The next step is to test the seven-checkpoint protocol with more first-time learners and use their feedback to improve explanations, checkpoint pacing, and accessibility. After that validation, ReproPath can explore additional curated NLP cases, guided custom intake, broader storage recovery, and wider cross-browser coverage while preserving the same source, evidence, provenance, and claim-boundary requirements.
Built With
- github-actions
- gpt-5.6-thinking
- openai-codex
- react
- react-router
- typescript
- vercel
- vite
- vitest
- zod