DocDrift

Text description

What DocDrift does

README code examples quietly rot: an argument gets renamed, an import moves, a dependency drops an overload, and the docs keep lying. DocDrift finds the snippets in your README and docs that no longer match your actual code, asks an NVIDIA Nemotron model on Nebius Token Factory to propose a fix, and then re-runs each proposed fix in a sandbox - so only verified fixes are ever reported.

Why the rerun matters

AI fix suggestions look convincing and are often wrong. DocDrift's core rule: a fix is only a fix if it executes. In the live hackathon test run, Nemotron proposed two fixes - one passed its rerun and was reported as verified; the other failed its rerun and was rejected; that run's rejected code and rerun error were not retained. That rejection is the product working, not failing.

How it works

Extraction. Every runnable snippet from README.md, README.rst, docs/*.md|.rst: Markdown fences, RST code-block directives, pycon fences, bare >>> doctest blocks - normalized to runnable code.
Execution. Docs build state across blocks, so snippets run in reading order with a shared namespace, the way a reader follows them. Execution runs inside bubblewrap: no network, read-only repo bind, no host credentials visible, CPU/memory/time limits.
Honest classification. Before anything is called "stale docs," failures are classified: environment (missing deps), service (needs a live DB/network), cascade (a name an earlier failed snippet would have bound). A model reply with no parseable fix is recorded as a generation/format outcome, never as a fix evaluation. Report categories are disjoint - executed = evaluated + environment + service + cascade.
Fix and verify. Genuine stale candidates go to Nemotron. The proposed fix reruns in the same sandbox. Pass: reported with the verified code. Fail: rejected with the rejection error.
Output. A JSON report (per-snippet evidence, model provenance, measured token usage) and a self-contained HTML page showing original snippets, verified fixes, and rejected fixes with their rejection reasons - and a banner stating plainly whether the report is a live run or an offline dry run.

How it uses Nebius and NVIDIA

All model calls go through the Nebius Token Factory OpenAI-compatible API using NVIDIA Nemotron open models (tested live on Nemotron-3.5-Lightning; tier choice is one model-id change). The client records provider, model and token usage per run, sends a per-call output cap on every request, and supports a run-wide token budget that refuses calls before they would cross it. The guard reduces overspend risk; it is not a billing guarantee.

Where it stands (measured, not promised)

65 automated tests passing; public repo, MIT license, zero dependencies for offline use.
Live on Token Factory (Oct 29): fixture pipeline verified end to end - correct failure diagnoses, 1 verified fix, 1 rejected fix, measured cost $0.0013 for the 4-call run.
Honest limits: the live evidence covers small synthetic fixtures, not real open-source repos yet; a real-repo live benchmark, a GitHub Action for PR-time doc checking, and a hosted demo are the next steps.