-
-
homepage and branding.
-
successful shipping repair and test results.
-
callback defect and passing reference control.
-
matched documentation quote and Tavily evidence.
-
custom Python input.
Inspiration
Hey there, I'm Fouad and here to talk about what inspired me to build AgentArena. So I noticed that coding agents can produce convincing explanations and pass an existing test suite while leaving the reported bug unresolved. But an AI-generated test can also be wrong. That creates two questions: does the implementation satisfy the requirements, and do the tests make valid claims?
I built AgentArena to make that review inspectable. Instead of asking developers to trust a summary, it gives them the patch, observed tool actions, generated tests, execution results, and evidence behind the verdict.
What it does
AgentArena is a working verification workbench for developers reviewing AI-generated Python fixes.
It supports two workflows:
- Repair + verify: NVIDIA Nemotron investigates a demonstration repository, attempts a production-code fix, and runs the existing tests. A separate verifier generates additional tests that challenge the resulting implementation.
- Audit only: the verifier challenges existing code without modifying it.
Developers can choose from six built-in scenarios covering discounts, authentication, inventory, shipping, redirects, and payment callback decoding. They can also paste their own single-file Python module, describe its expected behavior, and optionally supply pytest tests.
Each investigation exposes the evidence: patch diff, test source, failure output, research references, and execution log. Built-in investigations are saved in run history. Custom-code reports are downloadable and are not stored in shared history.
How I built it
The frontend uses Next.js, TypeScript, and Tailwind CSS. A Python FastAPI backend coordinates the coding agent, research, verification, scoring, and report storage.
NVIDIA Nemotron runs through Nebius Token Factory. The deployed model is nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B. Its coding-agent role uses bounded repository tools to inspect files, modify permitted production source, and execute tests. Its verifier role generates additional pytest checks.
Token Factory provides hosted inference for both roles, letting the project run without hosting model weights or maintaining a GPU server.
Tests execute in temporary Docker environments with an unprivileged user, disabled networking, a read-only source mount, and CPU, memory, process, and execution-time limits.
The interface is deployed on Vercel. An Oracle Cloud Arm VM runs the backend and Docker, with a persistent ngrok HTTPS tunnel connecting the two. Model inference remains on Nebius Token Factory.
Testing the tests
A failing generated test is not automatically a valid defect.
For built-in scenarios, AgentArena runs the same generated suite against a curated corrected implementation withheld from the agents. If that control fails, the platform returns an inconclusive result rather than treating the generated expectations as established facts.
Custom submissions have no corrected reference. Their reports therefore distinguish reproduced failures from passed checks and explicitly state that the results remain unvalidated.
The test-pass score summarizes executed checks. It is not a probability that the code is correct or secure.
How Tavily contributes
Tavily searches for supporting documentation and, when needed, extracts the requested primary documentation page. Retrieved text is treated as untrusted reference material, not as instructions or a replacement for the developer’s requirements.
The payment callback scenario demonstrates this workflow. Its original tests passed, but the helper discarded form fields with empty values. Tavily retrieved Python 3.11 documentation explaining that parse_qsl ignores blank values by default.
Nemotron generated tests that exposed the missing fields. Two of three generated tests failed against the buggy implementation, while all three passed against the corrected reference. The report connects a matched documentation excerpt to the test rationale and executed failure.
Excerpt matching checks provenance; it does not independently prove that an assertion is semantically correct. The current evidence demonstrates a useful research-informed workflow, not a measured improvement over a baseline without research.
Results from the deployed application
In a completed shipping investigation, the coding agent changed weight < 1 to weight <= 1, correcting the inclusive one-kilogram boundary. The patch passed all four original tests, all three generated tests, and all three corrected-reference checks.
The callback audit reproduced data loss despite the original suite passing. A redirect audit also exposed a production exception for non-string input and a failure to reject invalid characters, with the generated suite passing the corrected reference.
These are individual functional demonstrations on curated scenarios, not a general reliability estimate.
Challenges I faced
One challenge was preventing the verifier from inventing requirements. For example, a generated callback test treated consecutive separators as an additional empty field. The corrected reference also failed that expectation, so AgentArena correctly withheld a defect verdict.
Another was making documentation extraction robust to changing Markdown formatting without substituting unrelated passages or fabricating citations.
Deployment exposed a Linux permission issue: Python’s private temporary directory could not be read by the container’s unprivileged user. I fixed it by mounting a readable child workspace while retaining a private parent directory on the host.
The coding agent also needed clearer boundaries. Test files remain protected from modification, passing baseline tests do not resolve an uncovered bug, and an empty model response receives a bounded recovery attempt.
What I learned
The verifier needs verification too.
A polished explanation is less useful than a reproducible failure. A source link is less useful than a relevant passage connected to an executed test. And an inconclusive result can be the correct outcome when the available evidence does not support a stronger claim.
What’s next
The current product supports curated demonstration repositories and single-file Python audits using the standard library and pytest.
The longer-term direction is to evaluate externally produced patches through pull-request checks and CI integrations. Repository imports, dependency installation, and external-agent connections are future work. Broader use will also require stronger isolation and more extensive evaluation before accepting arbitrary repositories.
Log in or sign up for Devpost to join the conversation.