AgentOps AI

AI incident engineer that investigates Python errors, generates and tests fixes, and opens GitHub pull requests with human approval. Powered by NVIDIA Nemotron on Nebius.

Comment

🛠️ AgentOps AI: From Python Errors to Verified, Human-Approved Fixes

💡 Inspiration

A production error is only the beginning of an engineer’s work. Before opening a pull request, someone must investigate the logs, locate the relevant code, understand the root cause, reproduce the bug, write a fix, and check that nothing else breaks.

For small engineering teams, this repeated investigation takes time away from building their product. During an incident, it also means moving between monitoring tools, repositories, terminals, and code reviews while under pressure.

We built AgentOps AI to bring that workflow into one engineering workspace: an AI system that investigates an error, proposes a focused repair, and produces test evidence that a developer can review.

Our central principle is simple: a proposed fix should earn trust through evidence, and a human should control whether it becomes a pull request.

The Nebius × NVIDIA Global AI Hackathon gave us an opportunity to explore this using NVIDIA Nemotron models served through Nebius Token Factory.

⚙️ What it does

AgentOps AI turns Python application errors into reviewable code fixes through a coordinated pipeline:

Detect → Investigate → Fix → Verify → Human approval → Pull request

🔍 Detect and organize incidents

The system accepts errors through raw logs, uploaded log files, a drop-in Python exception handler, or Sentry webhooks.

It parses Python tracebacks, groups repeated errors, assigns severity, and checks whether an error resembles a previously resolved incident.

🧠 Investigate the actual repository

The Investigator uses NVIDIA Nemotron reasoning with repository tools to:

  • Read relevant source files.
  • Search for related code.
  • Inspect git history.
  • Consult similar past fixes.
  • Use optional web search for supporting context.

It produces a root-cause explanation grounded in the repository and the reported error.

🩹 Propose a focused repair

The Fixer proposes a minimal Python patch and a new regression test intended to reproduce the original bug. The application constructs the code diff from the proposed edits.

Patch sizes and editable paths are constrained, and existing tests and protected configuration files cannot be changed through the patching workflow.

🧪 Verify the fix with before-and-after evidence

Under the current default verification policy, a patch is considered verified only when:

  1. The new regression test fails on the original code.
  2. The same test passes after applying the patch.
  3. Previously passing tests continue to pass.
  4. Previously completed tests still complete.
  5. Both runs finish without collection errors or unsuccessful runner exit codes.

This prevents a generated patch from being accepted simply because it looks plausible or because a test disappeared from the results.

🙋 Keep the developer in control

The dashboard presents the diagnosis, proposed diff, regression test, verification results, and audit trail.

A human can approve or reject the proposal. Approval triggers GitHub pull-request creation; AgentOps AI does not automatically merge it.

🔁 Continue through code review

Authorized reviewers can request changes using /agentops <instructions> or a GitHub review. The system can revise the patch, repeat verification, and update its PR branch.

When a PR is merged, the fix becomes part of the system’s stored incident memory. A returning error can then be flagged as a potential regression.

🖥️ A complete developer workspace

The final frontend brings the workflow into a responsive charcoal-and-emerald interface. A public landing page introduces the project with an interactive 3D hero and a fallback for browsers without WebGL. Developers can sign in and move between:

  • Overview: a filterable incident inbox and fixes awaiting review.
  • Incident details: root-cause evidence, code diffs, regression tests, verification results, and an audit trail.
  • Repositories: register Python repositories and track clone and dependency setup.
  • Send a test error: submit an error to demonstrate the incident workflow.
  • Activity and metrics: follow agent actions, outcomes, timings, and token use.
  • Integrations, team, and settings: inspect connections, manage access, and configure workspace preferences.

Live incident updates use Server-Sent Events with polling fallback. A command palette supports Cmd/Ctrl + K navigation.

👥 Accounts, demo access, and persistent data

The application supports email/password accounts and optional GitHub OAuth sign-in. The first real account becomes admin; later accounts default to viewer. Admins can grant member access for repository actions and fix approvals.

A built-in demo account provides read-only access. Its viewer restriction is enforced by the backend, so visitors can explore without permission to register repositories, approve fixes, or manage users. Demo sign-in requires the running backend.

Accounts, registered repositories, incidents, and sessions are stored in the backend database, using SQLite by default with PostgreSQL configuration support. Added repositories remain registered after a browser refresh or backend restart when the database is retained. Passwords are stored as scrypt hashes in the backend; browser storage holds session tokens, connection settings, and preferences.

Configurable email, Slack, and webhook notifications keep developers informed as incidents progress.

🛠️ Final project improvements

We refined the frontend into a complete engineering workspace with a 3D landing page, responsive navigation, incident evidence views, and a test-error form. We connected repository registration and account access to the backend so saved records remain available after refresh when the database is retained. We also added backend-enforced read-only demo access, team roles, live incident updates, and clearer setup documentation.

We documented our repair demonstrations in a separate test repository, including before-and-after verification, generated regression tests, human approval, and reviewer-requested revisions.

🏗️ How we built it

We built AgentOps AI around a stateful orchestration layer that coordinates specialized agents, verification tools, and explicit approval transitions.

🤖 NVIDIA Nemotron on Nebius Token Factory

Our configured model roles are:

  • NVIDIA Nemotron 3 Nano: incident summarization within the detection stage.
  • NVIDIA Nemotron 3 Ultra: repository investigation and patch generation through tool calling.

Nebius Token Factory provides the OpenAI-compatible inference endpoint. This lets the backend use hosted NVIDIA models while managing tool execution, workflow state, retries, and approval rules in our own application.

🧩 Specialized responsibilities

The pipeline separates four core responsibilities:

  • Detector: parses errors, groups incidents, and identifies possible regressions.
  • Investigator: explores repository evidence and submits a root-cause diagnosis.
  • Fixer: proposes bounded code edits and a regression test.
  • Tester: executes pytest and evaluates before-and-after results using JUnit XML.

The Tester is a deterministic verification component. Its decision depends on test outcomes rather than a model’s assessment of its own patch.

🖥️ Application stack

  • Frontend: React 18, TypeScript, Vite, React Router, Three.js, and React Three Fiber.
  • Backend: Python, FastAPI, SQLAlchemy, and Pydantic Settings.
  • Persistence: SQLite by default, with PostgreSQL configuration support.
  • Live updates: Server-Sent Events with frontend polling fallback.
  • Access control: backend-enforced viewer, member, and admin roles; session-based accounts; optional GitHub OAuth.
  • Verification: pytest, JUnit XML, and Docker or subprocess execution.
  • Integrations: GitHub REST API and webhooks, Sentry, SMTP, Slack, and optional Tavily search.

🔐 Controlled execution

The model receives specific tools rather than unrestricted shell access.

Docker verification runs without network access and with resource limits. The subprocess option uses a scrubbed environment and a timeout, although it provides weaker isolation than Docker.

Additional controls include path confinement, protected files, signed webhooks, reviewer authorization, retry limits, token budgets, and audit records.

🧗 Challenges we ran into

Proving that a fix addresses the reported bug

A passing test suite alone does not establish that a patch repaired the incident. A generated test could pass before the fix, or an unsuccessful test run could produce misleading results.

We addressed this with explicit before-and-after verification: the new test must expose the original failure and pass after the repair, while preserving existing test behavior.

Turning model output into controlled code changes

Repository content, logs, and review comments are external inputs. They also contain information the agents need to investigate.

We structured tool access, marked external content as untrusted data, constrained patch paths and sizes, and protected existing tests and configuration files. These controls help keep model reasoning within the application’s workflow.

Enforcing approval in the workflow

Human approval needed to be an application rule. We separated verification from PR creation and introduced an explicit approval state, with recorded approval or rejection decisions.

Coordinating a complete incident lifecycle

An incident can require retries, reach the attempt limit, receive review feedback, have its PR closed, or return after a previous fix was merged.

Handling these transitions required a state machine and persistent history so the system could represent outcomes beyond the successful demo path.

Connecting the redesigned frontend to persistent backend state

The interface needed to represent real repositories, incidents, and permissions. We connected the workspace to backend APIs, retained data in the database, and made the demo account read-only at the server level. This lets the same application support a visitor tour and an authenticated engineering workflow.

Separating automated checks from live integrations

Local tests provide repeatable coverage, while real model calls and external services introduce credentials, network conditions, and service-specific behavior.

We document automated checks separately from historical live demonstrations so readers can understand what each type of evidence establishes.

🏆 Accomplishments that we're proud of

  • A complete incident-to-PR workflow: connecting error intake, repository investigation, patch generation, test verification, and human approval.
  • Evidence-based verification: requiring a regression test that fails before the repair and passes afterward.
  • A redesigned developer workspace: a public 3D landing page and a responsive dashboard bringing diagnosis, diffs, test results, incident history, and approvals into one interface.
  • Persistent accounts and repositories: backend database storage keeps registered repositories and incident history available after refresh.
  • Controlled demo access: a built-in viewer account lets visitors explore while backend roles protect write operations.
  • A review feedback loop: supporting authorized GitHub revision requests and re-verification.
  • Practical NVIDIA model integration: assigning different model roles through Nebius Token Factory.
  • A documented backend test suite: coverage includes workflow behavior, backend regressions, authentication, roles, webhooks, and local HTTP integration scenarios. Earlier validation recorded 70 tests; the latest environment run collected 71, with 49 passing and 22 blocked during HTTP fixture setup by the environment’s SOCKS proxy configuration.
  • A successful frontend production build: the updated React and TypeScript frontend builds with Vite. Browser visual verification and a fresh live external-service run remain separate checks.
  • Recorded live demonstrations: the repository documents Nemotron-generated fixes, GitHub PR creation and revision, a Sentry-to-PR flow, Docker verification, and delivered email notifications.

The automated end-to-end checks use a real local HTTP server and git repository, with scripted model responses and mocked external APIs. The documented live runs provide separate historical evidence of connected-service behavior.

🧪 Testing and public repair evidence

We used AgentOps Test Shop, a small FastAPI orders service, as the target repository for our repair demonstrations. The table below reproduces the historical results recorded in its public pull requests.

Test / PR Error and change Before: passed / failed After: passed / failed New failures Runner
PR #1 Missing discount caused a TypeError; default discount rate changed to 0.0 2 / 4 3 / 3 0 Subprocess
PR #2 Serializer caused a KeyError; read customer_ref while preserving customer_id in the response 2 / 4 3 / 3 0 Subprocess
PR #3 Pagination caused an IndexError; corrected one-based indexing and guarded first-item access 2 / 4 4 / 2 0 Subprocess
PR #4 Empty orders caused ZeroDivisionError; added an empty-list guard and regression coverage 3 / 5 4 / 4 0 Subprocess
PR #5 Serializer repair verified with two newly generated regression tests 2 / 6 5 / 3 0 Docker

All five PRs record first-attempt verification and human approval. Each row is an independent run against its own analysed commit; unrelated failures remain and the counts should not be combined into one suite result.

Additional check Recorded evidence
New regression tests in PR #5 Two new tests fail on the original code and pass after the repair
Reviewer feedback in PR #4 An /agentops request removes an unused pytest import; a follow-up comment reports re-verification and a revision commit is visible
Sentry-related demonstration PR #5's approval note identifies the fix as coming from a Sentry alert; original webhook delivery logs are not included in the PR
Default-branch target tests Six API tests cover health, discounted and undiscounted orders, serialization, first-page and last-page behavior
Frontend production build React/TypeScript and Vite build passed in the recorded local validation
Main backend local test run 71 collected: 49 passed; 22 HTTP fixture setup errors caused by environment SOCKS proxy support

PRs #1–#3 document repairs using existing tests; their descriptions do not show added regression tests. PR #4 records an empty-list test that fails before and passes after, while PR #5 records two such tests. This distinguishes earlier runs from the current default regression-test policy.

The five PRs were open and unmerged when reviewed on 30 October 2026. The test-shop default branch retains its demo bugs, and new regression tests are on the corresponding PR branches. These public records supplement the separate main-backend checks; they are not a fresh rerun or an all-tests-passing claim.

🎓 What we learned

Reliable agentic engineering depends on the surrounding system. Model reasoning becomes useful when tools, state transitions, verification, and recovery behavior are carefully defined.

We learned that regression tests are valuable evidence: demonstrating the failure before applying the patch makes a repair easier to assess.

We also learned that human oversight works best when the reviewer receives a concrete package: the diagnosis, code diff, test evidence, and action history.

Using different model roles encouraged us to consider where deeper reasoning is necessary and where a smaller model can support the workflow.

Finally, we learned to describe validation precisely. A local integration test, a live API demonstration, and a production deployment establish different things, and clear documentation helps developers judge the system fairly.

🚀 What's next for AgentOps AI

Our next steps focus on broader coverage and stronger operational reliability:

  • More monitoring connectors: add CloudWatch and Loki ingestion.
  • More programming languages: extend beyond Python with language-specific patching and test verification.
  • Durable background execution: introduce a persistent job queue for incident processing.
  • Multiple-worker support: move shared session and rate-limit state beyond a single backend process.
  • Database migrations: add Alembic for controlled schema changes.
  • Broader evaluation: measure repair success, regression-test quality, latency, and token usage across more repositories and bug types.
  • Deployment hardening: improve secrets management, operational monitoring, and isolation for hosted use.

AgentOps AI is currently a hackathon-stage engineering prototype. Its effectiveness depends on repository context, test quality, model behavior, and the configured execution environment.

Our goal is to help developers spend less time reconstructing failures and reach a well-supported review decision sooner.

Built With

Share this project:

Updates

Submission history