Anchor

An on-call agent that can't lie to itself. Its memory and its actions commit in the same transaction, or neither happens.

Comment

Anchor — An On-Call Agent That Can't Lie to Itself

An autonomous agent where memory and actions commit together—or neither does.

Inspiration

Imagine an AI agent responding to a production incident. It needs to create a SQL user to restore a service. The request reaches the infrastructure, the operation succeeds, but the connection drops before the agent receives confirmation.

What happens next?

If the agent retries, it might repeat the operation. If it assumes success, it risks recording something that never happened. Either way, its memory can diverge from reality, and every decision that follows inherits that uncertainty.

Most agent-memory frameworks focus on storing information and retrieving it later. But reliable autonomous systems need something more fundamental: the ability to ensure that what an agent remembers about its actions matches what actually happened.

That question inspired Anchor, an autonomous on-call agent for CockroachDB Cloud. We wanted to explore a difficult problem in agentic AI: how can an agent take real-world actions, survive crashes and network failures, and maintain a trustworthy history without blindly repeating potentially destructive operations?

Our central insight was that exactly-once execution cannot be achieved through retries alone. It requires a durable intent record, a verifier that can establish whether the intended operation occurred, and a protocol that preserves the integrity of the agent's memory.

What It Does

Anchor is an autonomous on-call agent that diagnoses incidents, executes operational actions through the ccloud CLI, and learns from recurring incident patterns.

Unlike a conventional agent that simply records an action after executing it, Anchor treats intent, execution, verification, and memory as parts of a reliability protocol.

Its core capabilities include:

  • Durable intent tracking: Records what the agent intends to do before it interacts with external systems.
  • Crash-safe execution: Handles failures between an external operation and the agent's recording of its outcome.
  • Evidence-based verification: Uses CockroachDB Cloud audit records to determine whether a specific intent actually caused an operation.
  • Three-valued outcomes: Classifies operations as Applied, Not Applied, or Unknown rather than inventing certainty.
  • Transactional memory: Commits the intent resolution, resulting world state, and 1024-dimensional embedding in one serializable transaction.
  • Reusable action protocols: Demonstrates that the same reliability interface can support other domains, including payment refunds.

The defining principle is simple: when the evidence is insufficient, Anchor escalates to a human instead of guessing.

How We Built It

We designed Anchor around three phases: Intend, Execute, and Commit.

1. Intend — Record Before Acting

Before making an external call, Anchor writes a durable intent record. Each intent is identified using a length-prefixed hash, avoiding ambiguous concatenations such as ("ab", "c") and ("a", "bc").

This creates a persistent record of what the agent planned to do, even if the process crashes immediately afterward.

2. Execute — Perform the Real Operation

Anchor executes the intended operation through the ccloud CLI.

These operations are deliberately treated as non-idempotent. For example, creating the same SQL user twice is not assumed to be harmless. A retry is therefore not considered proof of reliability.

3. Commit — Resolve Reality and Memory Together

After execution, Anchor verifies the outcome and commits the intent resolution, resulting world state, and 1024-dimensional embedding in one serializable CockroachDB transaction.

This is the heart of the architecture. If the database transaction fails, those records cannot be partially committed. If the external operation may already have occurred, Anchor does not pretend that a database rollback also reverses the external action.

Instead, it relies on durable intent and independent verification to recover safely.

Attribution Over Observation

One of our most important design decisions was to distinguish evidence that an operation occurred from evidence that our specific intent caused it.

For SQL user creation, Anchor derives the username from the idempotency key. When the CockroachDB Cloud audit log contains that attributable identity, the agent has evidence connecting the operation to its intent.

Merely observing that the desired state exists is not enough: another actor might have created it.

Three-Valued Verdicts

Anchor uses three possible outcomes:

  • Applied — evidence establishes that the intended operation occurred.
  • Not Applied — evidence establishes that it did not occur.
  • Unknown — available evidence cannot establish either conclusion.

The third outcome is essential. An uncertain operation should not automatically be repeated or marked successful. When attribution is insufficient, Anchor escalates rather than fabricating a history.

Technology Stack

  • Go: Core agent and control protocol.
  • CockroachDB Cloud: Durable intent records, transactional state, and distributed vector indexing.
  • Amazon Bedrock: Embedding generation.
  • AWS Lambda: Agent execution and console serving.
  • ccloud CLI: Real operational interactions with CockroachDB Cloud.

Challenges We Faced

1. Row-Level TTL and Foreign-Key Conflicts

Our initial schema encountered a subtle issue: row-level TTL could be accepted at DDL time but fail during runtime deletion because of foreign-key constraints, producing SQLSTATE 23503.

We redesigned the lifecycle so that ephemeral memory can decay without deleting the durable record of what the agent actually did.

What changed: Memory can expire; operational history remains trustworthy.

2. Silent Vector-Index Performance Degradation

We discovered that an incorrect vector-index operator or a predicate that did not satisfy the required prefix conditions could cause queries to fall back to full scans without an obvious warning.

We added EXPLAIN regression tests to catch these performance failures before they reach production.

3. Tests That Passed Without Testing the Guard

One test appeared to validate our memory-decay guard, but the guard was never actually exercised. The test passed while proving nothing.

We exposed the weakness by deliberately removing the guard and observing that the test still passed. We then strengthened our testing approach with mutation testing for load-bearing assertions.

4. Infrastructure and Integration Limitations

Several practical obstacles required additional investigation:

  • pgx could not directly bind the VECTOR type through our initial integration approach.
  • Access to crdb_internal was restricted in CockroachDB v26.2.
  • The ccloud CLI authentication flow did not support our attempted service-account-key approach.
  • AWS Lambda Function URLs were blocked by default in our new AWS account.

We documented these integration issues and their findings in docs/feedback.md.

What We Learned

Our biggest lesson was that exactly-once execution is a property of a protocol and its verifier together, never the protocol alone.

We learned this by finding a flaw in our own delete verifier. It matched an audit entry by username, but that username was not derived from the intent key. The audit entry proved that a deletion had occurred, not that our particular intent had caused it.

We changed the verdict to Unknown.

That decision captures the philosophy behind Anchor: a system should preserve uncertainty when the evidence is incomplete rather than convert it into a false fact.

We also learned that reliability must be tested under the failures that matter. Passing unit tests is not enough when the central claim concerns crashes, concurrent operations, transactional boundaries, and recovery after an external side effect.

What We're Proud Of

We did not stop at proposing a reliability architecture. We tested its central claim against a control implementation of the standard approach.

Using the same incident, the same external API, and the same crash scenario, we compared Anchor against a conventional control agent.

Scenario Anchor Control
Crash between acting and recording 1 external operation 2 external operations
Memory write fails without a crash Partial memory commit is structurally prevented World state changes while memory remains empty

We also tested concurrency and process failure:

  • 24 concurrent agents attempted the same action.
  • Only 1 agent acted.
  • 0 lost updates occurred.
  • The operation completed in 355 ms.
  • 35 tests passed, including a test that terminates a real operating-system process mid-action.

Finally, we tested whether the architecture generalized beyond infrastructure operations. Our internal/payments implementation supports refunds through the same interface without changing the protocol. Under the same crash scenario, the system avoids paying a customer twice.

These results are evidence from our tested scenarios, not a claim that every arbitrary external API can guarantee exactly-once execution without suitable verification or cooperation.

What's Next

Anchor points toward a broader approach to trustworthy autonomous systems: agents that can operate independently while maintaining an auditable, evidence-backed history of their decisions and actions.

The next steps include expanding the range of supported operational actions, improving recovery workflows for ambiguous outcomes, and strengthening verification against real-world failure modes.

Our goal is not simply to build an agent that can take action. It is to build one that can explain what it intended, establish what actually happened, and acknowledge what it cannot prove.

Because an autonomous agent should never have to lie to itself to keep moving forward.

Built With

Share this project:

Updates

Submission history