RECON

Verification-First Bug Fixing Agent powered by Nebius and NVIDIA

Comment

Inspiration

AI coding agents can produce a plausible-looking patch and say a bug is fixed, even when the bug remains or existing behavior breaks. RECON was built around a simple principle: treat a generated patch as a proposal, and use test results as evidence.

What it does

RECON's CLI accepts a local Python project and a bug report. It runs a four-stage workflow:

  • Baseline: Run the project's existing tests in a temporary working copy.
  • Reproduce: Use the configured model to generate a pytest test that should fail on the original code. If the test still does not fail cleanly after at most one repair attempt, RECON stops with COULD NOT REPRODUCE before generating a patch.
  • Patch: Generate structured search-and-replace edits and apply them to a temporary working copy.
  • Verify: Run the reproduction test and existing tests again.

RECON reports VERIFIED only when the reproduction test fails on the original code, passes after the patch, and the existing tests pass after the change. This is evidence from the tests run, not proof about untested behavior.

CLI exit codes reflect the outcome: 0 (VERIFIED), 1 (UNVERIFIED), 2 (COULD NOT REPRODUCE), and 3 (ERROR).

Safety and integrity guardrails:

  • AST Safety Scanning: The generated reproduction test is parsed and inspected before execution to block dangerous imports or calls (os, sys, subprocess, socket, open, eval, exec).
  • SHA-256 Tamper Checks: All existing tests and test configurations are hashed before and after patching. Patches that modify tests or configuration are rejected.
  • Environment Scrubbing: Sensitive API keys and credentials are scrubbed from the pytest subprocess environment.

How to Try RECON

  • Interactive Browser Demo (Offline Fixtures): Visit the public Render demo at https://recon-agent-frbn.onrender.com. It runs only the bundled offline good and bad test fixtures, accepts no user code uploads, and makes zero live API calls (hosted on Render Free Tier; subject to a possible spin-up delay if inactive).
  • Testing on Another Python Repository (Live CLI): To test RECON on another repository, clone the public repository at https://github.com/pramath675/recon-agent, follow the CLI installation instructions, and configure your own Nebius Token Factory API key in .env. Then run python -m recon --project <path> --bug-report <file>. Live CLI runs connect directly to your own Nebius account and may consume your account credits.

How we built it

RECON is built with Python, pytest, and Gradio.

  • Live CLI: Integrates with nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B via Nebius Token Factory (https://api.tokenfactory.nebius.com/v1/) to synthesize reproduction tests and structured search-and-replace patches.
  • Temporary Working-Copy Verification: All edits and pytest executions run strictly in ephemeral temporary copies; the user's original repository files are never modified.
  • Public Interactive Render Demo: Built with Gradio to demonstrate RECON's verification loop in the browser. It replays bundled good and bad test fixtures offline with zero live model calls and does not accept user code uploads.

Challenges we ran into

  • Enforcing strict budget limits: The default configuration budget is four calls total (RECON_MAX_API_CALLS=4). Stage repairs and the later bounded empty-response retry share that budget; --max-calls 2 is an optional cap used for the benchmark evaluation. Because calls are counted before dispatching, spending a call on a repair or retry can leave too few calls for a subsequent stage, causing RECON to halt cleanly before exceeding the limit.
  • Handling historical model errors: In those two-call-capped benchmark runs, RECON stopped safely when it encountered the truncated reproduction response and the empty patch response. The token-bucket case stopped during reproduction generation because the response was truncated at the 1,500-token output limit, after one call. The slugify case's patch-generation call returned an empty response; it stopped after two calls, with no budget remaining for a retry.
  • Model routing resolution: The initial live test request returned a model-not-found error due to an incorrect model identifier. Configuring the exact catalog routing ID for NVIDIA Nemotron-3-Nano-30B-A3B on Nebius Token Factory resolved it.
  • Subprocess test safety: Inspecting generated test code with AST scans before execution and validating file hashes to prevent tests from modifying themselves or existing project test suites.

Accomplishments that we're proud of

RECON has a working CLI, an offline interactive web demo, and a documented five-case live evaluation against Nebius Token Factory:

  • Evaluation Outcomes: Across the five cases evaluated under the optional --max-calls 2 cap, 3/5 cases were VERIFIED overall. One case repeated the bundled pricing bug and is not an independent bug archetype; of the four additional cases, 2 were VERIFIED (semantic version comparison and progressive tax calculation) and 2 ended in ERROR without a usable patch (token-bucket rate limiter and URL slug formatter).
  • Reported Call and Token Usage: The five-case live run used 9 API requests total (within the 10-call aggregate cap). Across completed requests, the provider reported 6,814 prompt tokens and 5,218 completion tokens; usage for the single truncated request was unavailable from the provider.
  • Methodology & Context: Passing tests demonstrate empirical resolution of the reproduction test without regressing baseline tests, not mathematical proof of correctness. These five small, curated benchmark cases do not establish a general success rate across arbitrary repositories.

Security and Limitations

  • Not an OS security sandbox: Tests execute in an ephemeral temporary working copy with a scrubbed environment (NEBIUS_API_KEY removed) and timeouts. RECON does not provide OS-level network or filesystem isolation. For untrusted repositories, we recommend running RECON inside a container or virtual machine.
  • Human review: The reproduction test and patch are LLM-generated; developers should review both the test and the unified diff before trusting the verdict.
  • Evaluation scope: The benchmark suite comprises five curated cases (one repeated demo bug and four additional cases) evaluated under strict two-call caps, which provides initial empirical observations but does not represent a general success rate.
  • Offline follow-ups:
    • The deterministic current_time token-bucket variant was checked offline only and was not part of the live results.
    • The bounded empty-response retry feature was tested with offline mocked unit tests only; it has not been verified in a live Nebius run.

What we learned

A plausible-looking patch is not proof that a bug is fixed. Treating patches as proposals and enforcing automated verification against real reproduction tests and existing test suites provides concrete, repeatable evidence about the behavior covered by the tests.

What's next for RECON

Broader evaluation remains future work. Future efforts could evaluate RECON against larger external codebases, standardized multi-language benchmark suites (such as SWE-bench), and diverse test frameworks beyond pytest.

Built With

Share this project:

Updates

Submission history