backcast

AI SRE Incident Commander with production-grade agentic memory on CockroachDB + AWS temporal belief reconstruction, provenance, and transactional action leases.

Comment

Inspiration

Backcast started with a problem that comes up constantly in incident response.

An alert fires, an engineer restarts a service, the dashboard turns green, and the incident is closed. A few days or weeks later, the same problem appears again because the restart never fixed the underlying cause. It only relieved the immediate symptom.

When teams review an incident afterward, two questions are surprisingly difficult to answer.

What did we actually know when the decision was made?

By the time a post-incident review happens, everyone already knows the root cause. It becomes easy to judge an earlier decision using information that was not available at the time.

The second question is:

Was the action we took actually the best option?

Usually, there is no way to know because the alternatives were never executed.

That led to the idea behind Backcast:

What if an AI incident-response agent could return to the exact state of knowledge it had when a decision was made, replay the alternatives that were available at that moment, compare the outcomes, and learn from the result?

Backcast was built around that idea.

It lets an incident-response agent rewind its memory to a historical decision point, create alternative branches, simulate those branches, compare their outcomes, and store the resulting lesson for future incidents.

The workflow is:

Rewind → Fork → Compare → Learn

What it does

Backcast is a counterfactual incident-response system for AI agents.

After an incident is resolved, Backcast returns to the exact database timestamp at which the remediation decision was made.

It uses CockroachDB's AS OF SYSTEM TIME capability to perform a real historical database read.

That distinction is important.

Evidence created after the decision point is not simply removed with an application-level filter. It is unavailable to the historical MVCC read itself.

This prevents a replay from accidentally using information that the engineer or agent could not have known at the time.

Once the historical state is reconstructed, Backcast forks the incident into several possible remediation paths and runs each one through a deterministic incident model.

Each branch is evaluated using the same scoring model.

The score accounts for recovery, operational risk, cost, time to recovery, unnecessary actions, and whether the incident recurs.

For the main demonstration, Backcast produces:

Remediation Score Outcome Recovery Time Risk Notes
Roll back deployment 0.88 Fixed 120 sec 0.2 Best
Increase connection pool 0.818 Fixed 210 sec 0.3 Alternative
Wait -0.10 No fix 0 sec 0.0 Failed
Scale out -0.25 No fix 90 sec 0.2 Failed
Restart service -0.36 Recurs 60 sec 0.1 Actual action

The original action was to restart the service.

The restart temporarily relieved the problem, but it did not fix the underlying cause, so the incident returned.

Rolling back the deployment would have fixed the actual cause.

Backcast expresses this difference as decision regret.

For each branch, the score is:

$$ S =

\mathbb{1}[\text{recovered}]

\left( 0.10u + 0.20r + 0.04c + 0.20\min\left(1,\frac{t}{600}\right) + 0.30\mathbb{1}[\text{recurred}] \right) $$

where (u) represents unnecessary actions, (r) represents aggregate risk, (c) represents cost, and (t) represents time to recovery in seconds.

Decision regret is:

$$ R = \max_{b \in \text{branches}} S(b) - S(\text{actual}) $$

For the main scenario:

$$ R = 0.88 - (-0.36) = 1.24 $$

The rollback therefore scored 1.24 points better than the action that was actually taken.

Backcast then stores the resulting lesson in the agent's long-term memory:

For a deployment that reduced the database connection pool, rollback-deploy is the strongest simulation-backed remediation.

The next time a similar incident occurs, the agent can retrieve that experience.

Incident Commander

Backcast also includes an Incident Commander powered by Amazon Bedrock.

The agent runs through a real tool-use loop. It can retrieve similar incidents using vector search, record new evidence, update hypotheses, revise confidence levels, claim an action lease, and resolve an incident.

Historical belief reconstruction

Backcast stores the history of beliefs instead of keeping only the latest version.

This allows the system to show what the agent believed at an earlier point beside what it knows now.

In the demonstration, a hypothesis moves from 11% confidence to 87% after evidence connecting the incident to a deployment becomes available.

The historical view does not see that later evidence.

Fencing-safe autonomy

Backcast also demonstrates how multiple autonomous workers can safely compete for the same action.

In the demo, 20 workers simultaneously attempt to claim one remediation.

Only one becomes the current owner.

The winning worker is then deliberately crashed. Another worker takes over, and the original worker is later brought back.

The original worker cannot finalize the action because its fencing generation is stale.

This demonstrates an important distributed-systems property: an expired lease does not necessarily mean that the previous worker has stopped executing.

Safe takeover also requires preventing the stale worker from completing the operation.

Custom incident simulation

The browser interface allows users to create their own incident, define the cause, add possible remediation choices, and calculate decision regret using their own values.

This makes the counterfactual engine usable beyond the built-in demonstration.

How we built it

The main architectural decision was to make CockroachDB the system of record for the agent's memory.

CockroachDB is not simply a database sitting next to a separate AI memory system.

The agent's operational memory lives in CockroachDB.

Backcast organizes that memory into several layers.

Memory Type Stored Information CockroachDB Capability
Episodic Incident evidence VECTOR(1024) with C-SPANN vector search
Belief Hypotheses and belief history Bitemporal records
Semantic Verified facts Persistent structured records
Procedural Learned remediation knowledge Historical retrieval
Action Action leases and ownership Unique constraints and fencing generations
Provenance Event history and checkpoints Hash-chained ledger
Working Temporary agent state Row-Level TTL

Episodic memory

Incident evidence is stored as immutable records.

Embeddings are stored using VECTOR(1024) and retrieved with CockroachDB's C-SPANN vector index.

This allows the Incident Commander to find semantically similar incidents from previous runs.

Belief memory

Hypotheses and beliefs are stored bitemporally.

Records track when a belief became valid, when it stopped being valid, and which later belief replaced it.

This is what makes historical belief reconstruction possible.

Semantic and procedural memory

Verified facts and successful procedures are retained as reusable long-term knowledge.

A lesson discovered during one incident can therefore influence the next similar incident.

Action memory

Autonomous actions use leases backed by transactional database constraints.

Each action is protected by a uniqueness rule based on its organization and logical action key.

A fencing generation is also assigned to the current owner.

If ownership changes, older workers can no longer finalize the action using a stale generation.

Provenance memory

Important events are written into a hash-chained ledger.

Periodic ledger roots are signed using AWS KMS with ECDSA P-256.

This provides a verifiable history of what was recorded and in what order.

Working memory

Temporary agent state uses CockroachDB Row-Level TTL.

The database itself expires short-lived records rather than requiring a separate cleanup process.

Why keep everything together?

Historical reconstruction, vector retrieval, belief updates, provenance, and action ownership all operate against the same transactional system.

That avoids having one database for operational state, another vector database for memory, and another service for locks or action ownership.

It also reduces the possibility of different systems disagreeing about which version of the agent's state is correct.

AWS architecture

The deployed application is serverless and defined using AWS CDK.

Component Purpose
CockroachDB Transactional and temporal agent memory
AWS Lambda Application services
Amazon Bedrock Agent reasoning
Amazon Nova Pro Incident Commander model
Titan Text Embeddings v2 Evidence embeddings
API Gateway API ingress
AWS KMS Ledger checkpoint signing
Amazon S3 Artifact storage
Amazon EventBridge Scheduled consolidation
AWS Secrets Manager Database credentials
FastAPI Backend API
React Web application
TypeScript Frontend implementation
Python Backend and simulation
AWS CDK Infrastructure as code

Four ARM64 container-based Lambda services are built from the same image:

  1. ingest
  2. commander
  3. consolidate
  4. webapp

One architectural boundary was especially important:

The language model does not decide whether a remediation worked.

The model can analyze the incident and propose possible actions.

The result of each branch, however, is calculated by a deterministic incident model.

This means that running the same branch again produces the same result.

The comparison is therefore reproducible and auditable instead of depending on an LLM judging another LLM's decision.

Challenges we ran into

Cross-region database latency

The first deployed version of the counterfactual replay took approximately 16.5 seconds.

At first, we assumed the simulation itself was slow.

After instrumenting the database path, we found the actual problem.

One replay was making 64 database round trips.

The Lambda services were running in us-east-1, while the CockroachDB cluster was in ap-south-1.

With roughly 300 ms of cross-region latency per round trip, unnecessary database operations became the dominant cost.

We changed the implementation in several places.

Branch writes were grouped into a single transaction. Repeated inserts were replaced with multi-row inserts. The ledger gained an append_many path, and database connections were reused across warm Lambda executions.

The result was:

Metric Before After
Database round trips 64 20
Counterfactual replay time 16.5 sec 6.8 sec

The main performance problem was not CPU time.

It was network round trips.

Discovering that the headline feature was not fully wired

One of the most important issues we found was in the historical reconstruction path.

The application stored the HLC timestamp at which a counterfactual branch was created, and the documentation described Backcast as reconstructing the incident from that historical point.

However, the implementation was still reading the current version of the incident.

The timestamp existed, but it was not actually being used for the historical read.

We fixed this by introducing a real AS OF SYSTEM TIME query together with historical belief reconstruction.

We then added a regression test specifically for this behavior.

The test creates evidence after the fork timestamp and verifies that the historical reconstruction cannot see it.

That test is important because it verifies the central promise of Backcast directly.

Combining batching with a hash chain

The provenance ledger uses a hash chain in which each entry depends on the previous entry.

Conceptually:

$$ H_n = SHA256(H_{n-1} \parallel E_n) $$

where (H_n) is the current entry hash, (H_{n-1}) is the previous hash, and (E_n) represents the new event data.

This creates a challenge when trying to batch database operations because the hashes themselves are sequential.

The solution was to lock the current chain tip once, calculate the next sequence of hashes in memory, and write the resulting entries using a multi-row insert.

We then tested combinations of individual and batched writes and reverified the complete chain afterward.

Handling stale workers safely

An expired lease only proves that a worker stopped renewing it.

It does not prove that the worker stopped running.

Consider the following sequence:

  1. Worker A claims an action.
  2. Worker A stops renewing its lease.
  3. Worker B takes over.
  4. Worker A becomes active again.
  5. Both workers attempt to finalize the action.

Allowing both workers to continue would make autonomous remediation unsafe.

Backcast prevents this using fencing generations.

When Worker B takes ownership, it receives a newer generation.

If Worker A returns, its older generation is rejected.

The demo deliberately reproduces this failure scenario to verify that takeover works correctly.

Defining the limits of exactly-once execution

We initially considered describing the system as providing exactly-once remediation.

That would not have been technically accurate.

A CockroachDB transaction cannot atomically commit together with an arbitrary external AWS side effect.

Backcast therefore makes a narrower guarantee:

There is one current logical owner of an action and one canonical action intent.

Executing external remediations is currently outside the scope of the project.

Being precise about this limitation made the architecture easier to reason about and the technical claims more defensible.

Accomplishments that we're proud of

The biggest accomplishment is that Backcast goes beyond using an LLM to summarize logs or recommend an action.

It combines AI reasoning with temporal database reads, deterministic simulation, distributed coordination, vector retrieval, historical beliefs, and verifiable provenance.

Historical replay without hindsight leakage

Backcast reconstructs the database at the actual historical timestamp rather than asking the application to manually filter future information.

Evidence that did not exist at the decision point is not available to the historical read.

Making decision quality measurable

Backcast does not simply state that another remediation might have been better.

It runs alternatives through the same deterministic model, gives them comparable scores, and calculates the difference between the best available action and the action that was actually taken.

That difference becomes decision regret.

Fencing under real contention

The concurrency demonstration races 20 workers for the same logical action.

It then deliberately crashes the winner, allows a standby worker to take over, revives the previous owner, and verifies that the stale worker can no longer finalize the action.

Improving the deployed system with measurement

The counterfactual replay path was reduced from 16.5 seconds to 6.8 seconds.

That improvement came from measuring the actual database behavior and reducing round trips from 64 to 20, rather than guessing at the source of the slowdown.

Testing the project's main claim

When we discovered that the historical timestamp existed without actually controlling the incident read, we corrected the implementation rather than hiding the issue.

We also added a regression test proving that information recorded after the decision point remains invisible during historical reconstruction.

That test directly validates the main idea behind Backcast.

What we learned

One of the biggest lessons from Backcast was that AI memory is not only a model problem.

The way memory is stored determines what an agent can reliably reason about later.

Once evidence, beliefs, actions, and provenance have transactional and temporal guarantees, it becomes possible to reconstruct what an agent actually knew at a particular moment instead of approximating that state afterward.

We also learned how expensive unnecessary database round trips can become in a distributed system.

Reducing the number of database interactions produced a much larger performance improvement than optimizing the simulation code.

Another important lesson was that historical database reads provide a stronger guarantee than application filtering.

A query such as:

WHERE created_at <= decision_time

depends on every developer remembering to apply the condition correctly.

Using AS OF SYSTEM TIME makes the historical boundary part of the database read itself.

We also learned that distributed leases alone are not enough for safe autonomous actions.

If ownership changes, a previous owner must not only lose its lease. It must also lose the ability to finalize the operation.

That is the purpose of fencing.

Another lesson was to test product claims directly.

Instead of only testing individual functions, we added tests for statements the project makes to users.

For example:

Record evidence after the decision timestamp and prove that the reconstructed historical state cannot see it.

Finally, we learned that technical precision is more convincing than ambitious terminology.

Removing claims the architecture could not fully guarantee made the project stronger.

What's next for Backcast

Real remediation execution

The next major step is connecting the fenced action-lease system to a real remediation executor.

Before performing an external operation, Backcast would verify that the system is still in the state that justified the remediation.

This would allow the counterfactual and coordination infrastructure already in place to support controlled real-world actions.

Same-region deployment

Most of the remaining replay latency comes from communication between the Lambda deployment in us-east-1 and the CockroachDB cluster in ap-south-1.

Moving the database and compute closer together should reduce replay latency substantially without changing the counterfactual engine itself.

Multi-region resilience

If Backcast is going to help diagnose infrastructure failures, the agent's own memory layer should also survive regional failures.

Multi-region operation is therefore a natural next step.

Belief calibration

Because Backcast keeps a history of its beliefs and confidence levels, it can eventually answer another useful question:

When the agent said it was 90% confident, was it actually correct about 90% of the time?

That would make it possible to measure and improve the calibration of the Incident Commander rather than simply displaying confidence scores.

Incident auditor

A read-only MCP interface could expose the provenance and belief history to engineers or auditors.

For example, an engineer could ask:

  1. What did the agent know when this decision was made?
  2. Which evidence arrived afterward?
  3. Why was this remediation selected?
  4. Which alternative produced the highest simulated score?
  5. Where did this learned procedure come from?

The long-term goal for Backcast is to explore a more accountable form of AI agent memory.

An agent should not only remember what happened.

It should be able to reconstruct what it knew, explain why it acted, evaluate what it could have done differently, and use that experience to make a better decision next time.

Built With

Share this project:

Updates

Submission history