Fresh Eyes

Broken READMEs block new users. Fresh Eyes tests Python CLI setup, uses Nemotron to propose fixes, and verifies each repair twice in clean environments against human-approved checks.

Comment

Inspiration

A project's README is often a newcomer's first experience of the software. It promises a simple path: install these packages, run this command, and see a result. But a missing installation step can stop that journey immediately. A dependency can also change after the instructions were written.

Maintainers may never notice because their own machines already contain the package, command, or configuration that a new user is missing. We wanted to make that first experience testable.

Fresh Eyes follows the README in a clean environment, checks the actual result, and uses NVIDIA Nemotron to propose a small repair when the supported workflow fails. A repair earns the label verified_fix only after it passes the same approved checks twice from scratch.

What it does

Our MVP focuses on small, public Python command-line projects. It provides two levels of checking.

Installation and startup checks follow an unambiguous README installation command and a documented help or version command. These can run without a model key through the local application or GitHub Action. A passing result is labelled startup_only: it confirms that the documented startup path works.

Functional repair uses a stronger workflow:

  1. Pin the repository. Record an exact commit so the source cannot change halfway through a run.
  2. Define success before running. A person reviews the cited README lines, two sample inputs, and their expected outputs. The acceptance check is locked with a hash.
  3. Follow the original instructions. Run them in a fresh Linux container and compare the actual outputs with the approved expectations.
  4. Diagnose a failure. Nemotron classifies the problem and proposes a constrained patch to the README or permitted dependency declarations.
  5. Validate the proposal. Reject changes to product source code, acceptance checks, unsupported commands, or README text that does not match the cited lines.
  6. Verify twice. Run the patched instructions in a fresh container, then repeat in another independent clean container. Both must pass before the repair is marked verified.

The bilingual interface shows what ran, where it stopped, the proposed patch, both verification results, and the model-call ledger. Unsupported or ambiguous workflows receive an explicit explanation.

Fresh Eyes is also available as a reusable GitHub Action, pinned in our demonstration repository to v1.0.0. It runs supported installation and startup checks in fresh, non-root containers with PyPI-only network access. It needs no Nebius or Tavily key and makes no model calls.

Our demonstration contains two team-written versions of one small CLI. The working README passes. The broken README fails because installing only its requirements never creates the command. Both results were rerun using the released Action. These are engineering fixtures, separate from the two real AI repairs.

We also published a nightly rich-cli startup monitor, scheduled for 02:17 Europe/London. It checks the current default branch, pins the resolved commit, installs through PyPI, and runs the documented startup command. Reports include source/version information and checksums, with artifacts retained for 90 days.

The identical workflow passed real manual validation on 29 October. At submission preparation, the first automatic scheduled event is still pending. The public history distinguishes manual and scheduled triggers. GitHub schedules may be delayed or skipped. Nightly monitoring checks startup; it does not perform functional verification or automatic AI repair.

How we built it

We use NVIDIA Nemotron on Nebius Token Factory with two roles:

  • Nemotron 3 Nano classifies the observed failure from the execution evidence.
  • Nemotron 3 Ultra diagnoses the cause and proposes a small, structured repair tied to the relevant README lines.

The model returns structured JSON. A deterministic validator checks the proposal before any edit is applied. The acceptance checks remain under the control service, outside the repository's execution environment, and the model cannot rewrite them.

In selected real development runs, Tavily retrieved the project's installation documentation between classification and diagnosis. Search results are treated as untrusted supporting evidence and never define what counts as success.

Model requests use explicit budgets and a persistent spending ledger. Reported token usage determines the displayed cost, which remains labelled as pending billing reconciliation. Missing usage stays unknown.

The backend uses Python, a standard-library HTTP API, and SQLite for run records and append-only approval history. Execution uses isolated Linux containers. A controlled proxy permits package access to PyPI while restricting other outbound traffic.

The frontend uses HTML, CSS, and JavaScript modules with English and Chinese translations. A static build packages recorded evidence for GitHub Pages, allowing judges to inspect results without provider credentials. Its browser-only GitHub planner accepts a public repository URL, reads the README, and explains which installation/startup lines it would follow.

The frozen v1.0.0 source passed 403 Python tests and 49 frontend tests on an independent GitHub-hosted runner. Additional Docker checks verified the expected working and failing Action fixtures.

Challenges we ran into

Giving the model the right evidence. Early proposals correctly identified the missing installation but quoted README text that was absent from the supplied context. We changed context selection to prioritize the cited lines and added line-anchored edits. Invalid proposals remain rejected and visible.

Giving reasoning enough room. One Ultra request consumed its entire 1,536-token allowance in reasoning and returned no usable patch. A later 4,096-token request returned a complete proposal. We preserved these observations without presenting individual trials as a controlled model comparison.

Separating runner faults from README faults. An early environment mounted the workspace with noexec, preventing installed commands from starting. We retained the infrastructure failure, added a workspace probe, and excluded it from README repair claims.

Finding suitable real failures. Our selection rules screened roughly 3,600 repository observations. Many candidates needed credentials, system tools, services, or network access outside the runner's scope. Envdoctor also required two input files in the working directory, so we extended the runner while preserving the existing acceptance checks. Its N=1 selection was locked before execution but before the full scan finished; we disclose that deviation.

Accomplishments that we're proud of

We have verified AI-generated repairs on two distinct repositories.

md2html — verified run: The advertised README path installed requirements.txt, then asked the user to run md2html. Installation succeeded, but the command was never created because the project itself had not been installed. Nano classified the failure; Ultra changed the installation instruction to pip install .. The original instructions failed, the candidate passed, and the independent clean verification passed in the same pipeline, with no human patch edits or saved proposal reuse. This took two model calls and approximately US$0.0055 calculated from usage.

envdoctor — verified run: The first advertised instruction, pip install envdoctor, failed because PyPI had no distribution. We selected and locked this challenge case before executing it. Nano classified the dependency failure, and Ultra proposed pip install .. Both approved inputs passed in the candidate and independent verification runs. This took two model calls and approximately US$0.0036 calculated from usage. The README already contained a source-install alternative further down; the repair fixes the first advertised path and leaves a duplicated installation line.

These are development and challenge results, not a general success rate. Failed attempts remain visible alongside the successful repairs.

We are also proud of:

  • Human-approved, hash-locked acceptance checks that the agent cannot change.
  • Inspectable failure records, rejected proposals, model usage, and patches.
  • A released GitHub Action with a genuinely passing demonstration and an expected failing demonstration.
  • A published nightly startup monitor and a public history that distinguishes verified execution from a configured schedule.

What we learned

An installation completing successfully does not prove that a newcomer can use the software. Functional checks need to examine actual output.

A correct diagnosis also does not guarantee an applicable repair. Useful evidence, precise edit locations, deterministic validation, and independent verification all matter.

Finally, a clear refusal is valuable. When a project falls outside the supported workflow, the report should explain the limitation and preserve the evidence.

What's next for Fresh Eyes

The public website currently provides recorded runs and browser-only README planning. It does not execute newly submitted repositories. Real execution requires the local setup or GitHub Actions; live AI repair requires the owner's explicitly enabled provider key and budget.

Our four-case formal original-README baseline already passed all four cases. The B/C/D comparison groups have not run, so we claim neither a formal repair success rate nor routing savings. Two repaired repositories remain a small development sample.

Our next priorities are:

  • Broader functional scheduling beyond installation and startup checks.
  • Clearer comparisons showing which dependency versions changed between runs.
  • Maintainer notification options when a documented workflow stops working.
  • Support for more project types, including libraries and web services.
  • A larger, independently reviewed evaluation set.

These extensions will keep the same requirement: success must be defined independently of the repair agent.

Built With

Share this project:

Updates

Submission history