AgentGit

## Inspiration

AI coding agents are surprisingly capable — but they have no sense of when they're stuck. We watched agents loop through the same broken fix five, six, seven times in a row, each
iteration convinced it was making progress.

The deeper problem is that most agent safety work focuses on what the agent does —
filtering outputs, sandboxing execution. Almost nothing focuses on trajectory — the
pattern of decisions over time. The same bad step repeated
is a signal. We wanted to build the thing that reads that signal and acts on it.

## What it does

GitAgent is a self-correcting AI coding agent with a real-time safety layer. It takes a
broken Python codebase, attempts to fix it using Claude Haiku, and monitors its own behavior as it works.

After every step, an evaluator checks three failure conditions against live data in
ClickHouse Cloud:

Loop detection — consecutive agent thoughts are compared using Levenshtein similarity. Above 0.85, the agent is repeating itself.
GIVE_UP detection — if the agent explicitly signals it's stuck, the system acts
on it rather than ignoring it.
Stagnation detection — four consecutive failed steps with no progress triggers
rollback regardless of how varied the agent's reasoning appears.

When any condition fires, the workspace hard-resets to a git SHA snapshotted before the
run started, the conversation history clears, and the agent receives a plain-language
warning explaining why it was interrupted and what to do differently.

Every run streams live to an OpenUI dashboard — step trace, rollback events, ClickHouse
telemetry, and an AI Summary tab that generates a plain-English explanation of what
happened at each step. All runs are logged to MLflow for audit and comparison.

## How we built it

The core is a LangGraph StateGraph with three nodes: agent_think_and_act calls Claude Haiku and applies the fix; rollback_executor resets the workspace; clickhouse_eval is the conditional edge that decides which path to take.

Composio handles all file I/O and shell execution. Every file read and write goes
through FILETOOL, every pytest run through SHELLTOOL — so the execution environment is
fully swappable without touching the agent logic.

ClickHouse Cloud stores one telemetry row per step. The evaluator queries those rows
back rather than trusting in-memory Python state — making the decision layer stateless, crash-safe, and ready to work across multiple parallel agents.

Rollback runs git reset --hard <sha> against a commit SHA captured at startup, not
HEAD — so it lands on exactly the pre-agent state even if the agent committed intermediate
changes during a run.

The OpenUI dashboard is a FastAPI server with a Server-Sent Events endpoint that tails
the state log every 300ms and pushes JSON to the browser the instant a step completes.

## Challenges we ran into

Making loop detection actually fire reliably required careful problem design. A
single-function bug gets fixed in one step — no loop. The key was three functions with identical bugs tested with pytest's -x flag, which stops at the first failure. This
forces the agent to fix them one at a time, producing near-identical reasoning at each
step and reliably crossing the similarity threshold.

ClickHouse as evaluator source required careful sequencing — write the row, then
immediately query it back, then overwrite with the real decision. Getting the ordering
right so the evaluator was always reading its own freshly-written data took several
iterations.

## Accomplishments that we're proud of

I'm proud of the rollback design itself. It's not a retry — it's a deliberate
interruption with a diagnostic message. The agent doesn't just try again; it gets told
why it was stopped and what the correct strategy is. That distinction matters for
building agents that can actually learn from failures within a single run.

## What we learned

The safety layer has to live outside the agent. An agent cannot reliably detect its
own loops — it doesn't have the right vantage point. The evaluator needs to observe the agent from the outside, with persistent storage as its source of truth, not the agent's
own memory.

We also learned that agent failure modes are more varied than they first appear. Loops
are obvious, but an agent that produces slightly different broken approaches each step — never repeating, never progressing — is just as stuck. Stagnation detection covers the
cases that similarity metrics miss.

The hardest bugs in AI agent systems are infrastructure bugs, not model bugs. Claude
Haiku performed well throughout. The problems were in Composio's shell internals, Python version incompatibilities, and async I/O blocking. Building reliable agent infrastructure
requires the same rigor as building any distributed system.

## What's next for AgentGit

The current evaluator compares thoughts textually. The obvious next step is semantic
comparison — embedding-based similarity that catches loops where the agent uses different words to describe the same broken approach.

The rollback today resets to a fixed baseline. A smarter version would maintain
checkpoints at each successful milestone and roll back only as far as the last known-good state, preserving partial progress.

The infrastructure is already built for multi-agent use — ClickHouse stores a session ID
per row, and the evaluator queries by session. Extending this to a fleet of parallel
agents with a shared decision layer and coordinated rollbacks is the natural next
deployment target.

Finally, the current system is purpose-built for a debugging task. The evaluator
interface is generic — current_thought, previous_thought, exit_code, history — and the rollback mechanism is just a git reset. The same architecture applies to any
agent working in a version-controlled environment, from code generation to infrastructure
provisioning to document editing.