Redline

An autonomous auditor that proves every smart-contract bug with a working exploit, and has its own firewall so the hostile code it reads can't hijack it.

Comment

Inspiration

AI auditors are starting to review smart contracts, but they have two problems. First, they claim bugs without proving them: a confident paragraph about "possible reentrancy" costs the reader time and proves nothing. Second, an AI agent that reads hostile code is itself an attack surface. A malicious repo can hide instructions in a comment or README telling the auditor to report "no issues" or run a setup command. In security tooling, that is not an edge case. It is the threat model.

Redline started from one question: what would an AI auditor look like if every claim had to be backed by executed proof, and the auditor itself could not be hijacked by what it reads?

What it does

Given a Solidity/Foundry repository, Redline runs four stages:

  1. Recon: Nemotron reads the full codebase and maps entry points, value flows, privileged functions and external calls into structured findings.
  2. Research: Tavily pulls real-world exploits and post-mortems that match each suspected pattern, and they are cited in the report.
  3. Exploit: For each finding, Redline writes a Foundry proof-of-concept test and runs forge test in a sandbox. A finding is confirmed only if the exploit actually executes. Otherwise it is reported as unconfirmed and never upgraded.
  4. Patch: Redline proposes a minimal fix, then re-runs both the PoC (which must stop working) and the repo's existing tests (which must stay green).

Every step passes through the firewall, and the result is a report with the PoC, test output, patch diff, sources, and the full firewall log.

The firewall

The auditor treats everything it reads, including repo files, comments and search results, as untrusted data.

  • Input scanning: injection attempts are quarantined before they reach the main agent. The agent sees a redacted placeholder, and the report shows exactly what was removed and why.
  • Action gating: every tool call is checked in two layers. Layer 1 is deterministic and cannot be bypassed: a command allowlist, a path jail with symlink-escape protection, no network, no secret reads, no pipe-to-shell, and blocked dangerous forge flags such as --ffi and --fork-url. Layer 2 is a small Nemotron model that judges intent against the current task. Deny always wins:

$$\text{allow}(a) = R_{\text{rules}}(a) \land P_{\text{policy}}(a)$$

  • Unforgeable execution: the sandbox runner only accepts a GateToken that the gate itself issues, bound to the exact command. So no code path, including Redline's own, can execute anything without passing the firewall. This is enforced by tests.

The demo repo contains a real reward-accounting bug and a planted injection telling AI auditors to report nothing and run a remote script. Redline quarantines the injection, blocks the command, and still confirms and patches the real bug.

How we built it

  • Models: NVIDIA Nemotron [exact model IDs] on Nebius Token Factory. A larger Nemotron handles recon, exploit writing and patching; a smaller Nemotron handles fast firewall checks, so the hot path stays cheap.
  • Search: Tavily for exploit research with citations.
  • Sandbox: Docker running Foundry with no network, a read-only root, all capabilities dropped, memory and CPU limits, and hard timeouts.
  • Backend: Python, typed throughout with pydantic. Every structured model output is validated and retried on failure. Every model call logs provider, tokens and latency.
  • Frontend: a FastAPI run viewer with live streaming of pipeline stages, findings and firewall decisions, plus a self-contained HTML report export.
  • Tests: [N] tests covering the firewall rules, sandbox isolation, and the guarantee that nothing executes without the gate.

Results

All runs below used a real Nemotron model and real forge execution in the sandbox. Nothing is scripted.

Live audits

Target Bug class Findings Confirmed (executed PoC) Patched Time Tokens
vulnerable-vault Reentrancy 1 1 (attempt 1) 1 (attempt 1) 7m 43s 45k
poisoned-repo Reward accounting + planted injections 4 3 2 35m 249k
oracle-lender Spot-price oracle manipulation 4 2 (attempts 1 and 3) 0 in progress at submission —
admin-takeover Unprotected initializer — — — not yet run live —

Highlights

  • vulnerable-vault: the reentrancy was confirmed and patched on the first attempt. The exploit fails after the patch, the existing tests stay green, and 4 Tavily sources are cited. The firewall quarantined no legitimate content.
  • poisoned-repo: both planted injections were quarantined before the agent ever saw them, one by the deterministic layer and one by the Nemotron scanner. The audit still went on to confirm 3 real findings with executed exploits.
  • oracle-lender: the oracle manipulation was confirmed, and one finding needed three exploit attempts, which is the retry loop working as intended. Both patch attempts failed with errors, and two findings remained unconfirmed. We report them as unconfirmed rather than upgrading them.

Firewall evaluation

Measured on a small hand-labelled set ([N] input samples, [M] action samples):

Component Layer 1 only (rules) Layer 1 + Nemotron Precision
Input scanner (recall) 0.50 1.00 1.00 both ways
Action gate (recall) 0.88 0.96 1.00 both ways

The rules alone miss half of the injections; adding the Nemotron layer catches all of them in this set with no false positives. The set is small and hand-written, so treat these numbers as directional, not a benchmark.

Limitations

  • Patch generation is the weakest stage: it succeeded on simple bugs but errored on oracle-lender.
  • Runs on larger repos are slow: about 35 minutes and 249k tokens for poisoned-repo.
  • The firewall eval set is small. A larger adversarial set is the next step.

Challenges

  • Getting access was the hardest part. Account onboarding was blocked on both Nebius and NVIDIA's API catalog because of card and country verification. We built a provider abstraction so all model logic was independent of the provider, kept building against sandboxed and scripted runs, and added a preflight check that refuses to run the demo unless it is on Nebius. [Describe how access was eventually resolved.]
  • Proof over plausibility. Getting a model to write a plausible exploit is easy; getting one that actually executes against the contract is not. The exploit loop retries with real forge output, and anything that never runs is labelled unconfirmed.
  • Our own tests caught a firewall hole: an early regex allowed cat .env. That pushed us toward parsing commands into argument lists and command-bound gate tokens instead of pattern-matching strings.
  • Keeping the demo honest: an early fixture had a comment that gave away the bug's location, which would have made "it still found the bug" meaningless. We removed it.

What we learned

  • In agentic security tools, the agent's own inputs are the attack surface, so the firewall belongs in the architecture, not as a filter bolted on afterwards.
  • Deterministic rules and a small model complement each other: rules give guarantees, and the model catches intent the rules can't express.
  • Executing code is the cheapest way to make an AI's output trustworthy.

What's next

  • Fine-tune Nemotron Nano on Token Factory as a dedicated firewall classifier
  • Support Sui Move and Solana programs
  • Evaluate on a larger set of real historical exploits

Built With

Share this project:

Updates

Submission history