Inspiration
A README is a promise: follow these steps and you will reach a working build. The problem is that the people reviewing that promise usually already know the undocumented steps.
Greenhorn was inspired by a simple question: can a genuinely unfamiliar developer follow this repository’s documentation without filling gaps from prior knowledge?
What it does
Greenhorn audits a repository’s documented onboarding path through a literal newcomer.
It starts Codex in a fresh scratch session and lets it propose one plain command at a time. GPT-5.6 independently checks each proposal against the README and explicitly linked local documentation. Only a documented command can enter the isolated Docker executor.
Each report preserves the documentation quote, Codex’s proposal, GPT-5.6’s ruling, whether the shell received the command, the command result, and any verified checkpoint or time to first green build.
When the docs do not license a safe next step, Greenhorn stops and reports the exact documentation gap instead of guessing.
Current scope: Greenhorn evaluates Node.js repositories with a root package.json. Other runtimes return UNSUPPORTED RUNTIME by design, not a README-failure finding.
How we built it
Greenhorn is a Node.js CLI built in Codex. Codex is also the constrained newcomer runtime inside the product: it reasons from an empty scratch directory and proposes the next literal action.
GPT-5.6, through Azure AI Foundry or the OpenAI API, acts as the independent contract enforcer. It rules each proposed command and can draft a minimal README addition for supported documented halts. The checked-in pilot reports are Azure-backed; the OpenAI API is the fastest path for a fresh judge run when their API key has model access. (I have azure credits so I used azure foundry instead of OpenAI credits)
Accepted commands run in a pristine Ubuntu Docker container. Greenhorn generates static, inspectable HTML reports with pinned source revisions, transcripts, per-command rulings, and container evidence. GitHub Pages provides a zero-setup path for reviewing the checked-in pilot reports.
Challenges we ran into
The hard part was making “follow the docs literally” useful rather than rigid.
We needed to distinguish a command that is explicitly documented from one that merely feels obvious to an experienced developer. We also had to avoid turning an incomplete README into an excuse for the agent to improvise.
Another challenge was keeping the proof honest. A controlled fixture can demonstrate a complete doc-fix loop, but it should never be presented as an external-repository result. Likewise, a small pilot can show the mechanism working without becoming a claim about how common documentation failures are.
Docker bootstrap reliability was also a practical first-run concern. A runtime download failure is not a repository finding, so Greenhorn records it as a runtime error and documents the recovery path separately.
Accomplishments that we're proud of
We built an auditable loop instead of a black-box score.
Greenhorn’s checked-in public Node pilot includes pinned Docker, Codex, and GPT-5.6 runs. In the Express report, README-licensed npm install and npm test ran in fresh Docker; the documented test exited zero in 1.1 minutes, and the report marked the outcome LAST STEP COMPLETED.
We also demonstrated a complete fix loop on a clearly labeled controlled fixture: Greenhorn halted on an ambiguous instruction, GPT-5.6 drafted a one-line README clarification, and a disposable-copy rerun reached the documented checkpoint. The original repository is never modified.
What we learned
Documentation quality is easiest to trust when the decision trail is visible. Showing the source sentence, the proposed command, the ruling, and the shell receipt makes an audit useful to both maintainers and reviewers.
We also learned that model evaluation needs careful boundaries. The enforcer’s published precision and recall come from small labeled mechanism checks, including a disjoint held-out set; they are not presented as a broad performance claim.
What's next for Greenhorn
Next, we want to improve the first-run runtime preflight and error capture, expand beyond Node.js repositories, and keep improving the calibration set for edge cases in real documentation.
The core direction remains the same: help teams prove that a newcomer can reach a working outcome by following the documentation they ship.
Log in or sign up for Devpost to join the conversation.