Arbalest-Agentic Software Security

Inspiration

Finding a vulnerability is only the beginning. Developers still have to understand it, write a fix, and find out whether that fix actually works. Most scanners stop at a list of warnings. I built Arbalest to connect those steps in one developer workflow: find the bug, propose a patch, then attack the patch in an isolated sandbox before anyone trusts it.

What it does

Arbalest is an agentic security CLI and MCP server for code you own or are authorized to test. It runs on NVIDIA Nemotron (nvidia/Nemotron-3-Ultra-550b-a55b) through Nebius Token Factory, and runs generated attack code in Nebius Token Factory Sandboxes.

arbalest scan runs four phases:

1. Recon: a deterministic index of the source tree, languages, manifests, routes and dangerous sinks. --diff-base limits it to changed files.
2. Code analysis: five specialists run in parallel:
   • Nemotron reviews the source.
   • A regex secret scanner checks every file.
   • Optional Semgrep runs if it's installed.
   • A dependency check queries OSV for many lockfile formats (Python, npm/pnpm/Yarn, Cargo, Go, Composer, Bundler, Maven, NuGet), plus runtime end-of-life data.
   • A deterministic taint analysis traces source-to-sink flows in Python, JavaScript/TypeScript and Go.
3. Dependency check: the OSV results are folded into the findings.
4. Threat RAG: a local CWE knowledge base, scan memory and optional Tavily live search inform a final Nemotron analysis.

The result works as a CI gate:

arbalest fix has an Architect agent plan the remediation from repository evidence. A Translator agent then writes a reviewable patch. Your source files are not changed.

arbalest immune runs the wargame:

1. The Architect patches the finding.
2. A Red Team agent writes an attack harness and runs it against the patched code inside a disposable sandbox.
3. If the attack breaches the patch, the Architect reads the breach evidence and repairs it, and the next attack learns from that history.
4. The loop stops when one of these happens:
   • The patch holds with strong evidence: PATCH HELD.
   • The evidence is inconclusive, or too weak to certify (signature-only, or confidence below 0.7). Certification is blocked.
   • The iteration budget runs out. The patch is rejected.

A finding that holds is marked HEALED. Patches are staged in .arbalest/proposed-heals for human review and are only written to source with --deploy.

When a patch is healed after a confirmed breach, Arbalest writes an antibody: a Semgrep-style detection rule. It's kept only if it matches the vulnerable code and does not match the patched code. Later immune and fleet runs sweep for it, and scan passes it to the model as a known pattern.

Other features include probe (read-only live checks against a running service), a CycloneDX sbom, fleet mode across many repositories, and sealed scan archives with SHA-256 manifests that history and compare can read. There are 10 MCP tools, and SARIF, JSON and HTML reports.

How I built it

Arbalest is written in Python, with a Python SDK and a TypeScript SDK on the same engine.

• Models: every model-driven agent calls Nemotron through Nebius Token Factory. Pydantic validates each response before the next stage uses it.
• Sandboxes: I integrated the Nebius Token Factory Sandboxes REST API. Each run gets a disposable microVM with networking disabled, a wall-clock timeout, truncated output and cancellation. A run is rejected if the API reports networking as enabled.
• Proving isolation: arbalest sandbox-check runs a hello probe, then tries to connect to 1.1.1.1:53. That connection must fail. If the network is reachable, the check exits 1.
• Docker fallback: a hardened local Docker backend is available. It has no network, a read-only filesystem, all capabilities dropped, a non-root user, and pid, time and tmpfs limits.
• Budgets: scans run under explicit limits on time, model calls and cost, and they support checkpoints and resume.
• Testing: 687 automated tests. CI runs ruff, mypy, bandit and pip-audit on Python 3.12.

Challenges I faced

The hardest problem was making results trustworthy, not just plausible.

• Plausible but wrong model output. In live testing, a response was valid JSON but pointed to an entrypoint that wasn't in its own code bundle. Another failure came from giving the planning agent an empty architecture graph. Both needed better context and semantic validation; a bigger model alone didn't fix them.
• False clean results. A model provider that ignored a timeout could hand back a result that looked successful. In a security tool that's the worst failure, so every model call now fails closed when its deadline passes.
• Silent failures. A bug in how saved antibodies were formatted made the scan quietly skip them. Type checking caught it, and a regression test now guards it.
• Patches that hold can still break things. In my demo, the CWE-95 patch blocked the attack, but the Red Team harness warned that it broke legitimate literal parsing. That's why patches stay staged for human review instead of being applied automatically.
• Practical issues: Windows compatibility, unsafe output paths, inconsistent verification exit codes and misleading success messages. Provider refusals are respected rather than bypassed.

What I learned

Agentic systems need explicit contracts between stages. A successful API call doesn't guarantee a valid patch, and a generated patch doesn't show that a vulnerability is fixed. Running the attack in a sealed sandbox and recording the result is the most convincing evidence a tool can give, and when that evidence is weak, the right answer is to say so. Nemotron Ultra handled planning and patch generation well, but validation stays essential no matter how large the model is.

Accomplishments

• Live run on Nebius: Nemotron Ultra produced the findings and patches. The Red Team's attacks ran in live Nebius Token Factory Sandboxes, and each one shows up in the Nebius console as an auditable sandbox operation.
• Demo app scan: the scan reported command injection (CWE-78), code injection (CWE-94/95), SQL injection (CWE-89), hardcoded credentials (CWE-798), missing authorization (CWE-862) and improper input validation (CWE-20). Verdict: VULNERABLE, exit 1.
• Demo wargame: both critical findings were patched. The CWE-78 fix removed shell=True, and the CWE-95 fix replaced eval() with ast.literal_eval(). The Red Team's attack failed against both on the first iteration, so both were HEALED. Verdict: SECURED, exit 0. Patches were staged for review.
• Benchmarks: on the project's own benchmark fixtures, the deterministic layers detected 18 of 18 expected vulnerability classes. The taint analysis found 10 true positives with 0 false positives and 0 false negatives across 18 fixtures. These fixtures are self-authored; I haven't run external benchmarks yet.

What's next

• Baseline attack: run the exploit against the original code too, so certification proves the attack worked before the patch and fails after it.
• Tripwire rules in scans: run the 57 built-in deterministic tripwire rules and validated antibodies as real detection rules inside arbalest scan. Today they run only in immune, fleet and fix/verify.
• Deeper taint analysis: add SQL and path sinks for JavaScript, and model classes, closures and callbacks in Python.
• Sandbox resource limits: add CPU, memory and pid limits on Nebius sandboxes once the API exposes them.
• External benchmarks: run published benchmarks and report the numbers.

Built With

• chromadb
• docker
• httpx
• model-context-protocol
• nebius-sandboxes
• nebius-token-factory
• neo4j
• nvidia-nemotron
• pydantic
• pytest
• python
• ruff
• sarif
• semgrep
• tavily-api