Hindsight

Rewind what your incident agent learned.

Comment

Inspiration

Persistent memory makes an incident agent more useful, but it also makes mistakes durable. A stale runbook lesson can rank first during the next outage and repeatedly steer the agent towards the wrong operational recommendation. We wanted an SRE to be able to answer three questions: which memory influenced this decision, how can I correct it while retaining history, and did the correction materially change the recorded recommendation?

What it does

Hindsight is a governed-memory incident-response cockpit. It records the CockroachDB memory versions an agent recalled, the normalized CloudWatch observations it used, the bounded recommendation Gemini produced, and the human verdict on that recommendation.

The controlled payments replay holds the same incident input and normalized observations constant, with governed memory as the permitted intervention. A stale version of one belief is cited when Gemini records the scale_workers recommendation. The operator rejects that recommendation, previews the blast radius of a governed correction, and applies it. CockroachDB invalidates the stale version while retaining its history and reasserts the trusted version. Replaying the same incident input and normalized observations then records throttle_retries. Final hosted acceptance verifies this bounded causal chain. No infrastructure recommendation executes, and the replay does not prove service recovery.

The public comparison fails closed: Hindsight shows “recorded recommendation changed after correction” only when both structured recommendation fingerprints are valid, the incident input and normalized observations match, the governed-memory intervention is canonical, and durable correction lineage connects the two runs. It does not infer the result from recommendation prose.

How we built it

CockroachDB is both the memory substrate and the transactional system of record. Tenant-scoped vector retrieval selects relevant semantic memory. Serializable transactions bind the selected read to the incident run, decision, operator approval or rejection, governed correction, temporal lineage, and transactional outbox.

The application runs on AWS Lambda behind API Gateway and CloudFront. Amazon SQS carries durable agent commands, while EventBridge reclaims expired work. DynamoDB stores fenced realtime connection state. CloudWatch provides allow-listed, time-bounded diagnostic observations. Cognito protects operator controls while the accepted replay remains publicly inspectable. SSM holds runtime configuration, and S3 retains bounded evidence and lifecycle exports.

Gemini 3.1 Flash-Lite performs schema-constrained reasoning. The model can request only a server-owned diagnostic or return one bounded terminal recommendation; it cannot execute infrastructure changes. Gemini embeddings produce 1,024-dimensional vectors, partitioned by tenant, namespace, provider, model, and representation profile.

CockroachDB tools

Distributed Vector Indexing is on the runtime path. It ranks tenant-scoped memory for each incident run and records the selected memory version with the decision. DVI qualification, not Managed MCP, supplies the vector-plan evidence. Cloud Managed MCP Server was used by an earlier development-side audit client through get-table-schema and select-query under read-only OAuth inspection operations. The sanitized transcript reconstructs persisted decision, retrieval, memory, rewind, lineage, and embedding-profile identities without selecting memory content. It covers an earlier accepted signature run, not final-revision identity parity, and it is not the application's runtime connection. The recorded operations do not establish privileges for the runtime database principal.

Supplementary privilege evidence comes from a deterministic audit informed by pinned CockroachDB Skill SQL and executed through the restricted auditor role. It is separate from Managed MCP and is not presented as an autonomous Skill invocation.

AWS integration

AWS Lambda runs the API, worker, and realtime/changefeed components. SQS and EventBridge provide durable dispatch and reclaim; CloudWatch supplies the controlled diagnostic observations; Cognito and API Gateway enforce the public-viewer/operator boundary; DynamoDB supports fenced realtime delivery; SSM stores runtime configuration; S3 retains bounded evidence; and CloudFront serves the cockpit.

OpenTelemetry is exported through AWS ADOT to X-Ray with bounded sampling. Retained accepted observability run 31927724566 correlates API, dispatch, worker, memory-read, reflection, and memory-write spans. Realtime is separately traced and correlated by run and tenant; this evidence was reused rather than rerun for the final source revision.

Alert delivery is proven separately by final full acceptance run 31949390747 and artifact 9264362519: operational and budget subscriptions were confirmed, with challenge-bound ALARM → OK delivery to the encrypted controlled SQS receiver. That receiver is an inspection boundary, not proof of delivery to a human or on-call system.

Verified release revision ed9a027643cf043ec8e984e6a8b09784db457ab7 (v0.1.0) is bound to exact-main CI run 31949350803, full acceptance run 31949390747, aggregate provenance artifact 9264441112, and deployed verification run 31950313100. The repository’s later README-only documentation successor does not change that deployed release or the retained evidence.

Challenges

The hardest part was establishing causality without overclaiming. Different prose after a correction is not proof of a different operational recommendation. We introduced a deliberately narrow structured recommendation contract for the exact controlled scenario, bound it to a canonical fingerprint, and required invariant incident input and normalized observations plus one governed-memory intervention and durable lineage before rendering the comparison.

We also had to preserve history while keeping correction safe. The governed invalidation/reassertion uses an immutable preview with a fingerprint, expiry, namespace revision, and exact causal scope. The applying transaction rechecks those bindings, making the correction auditable and idempotent.

Accomplishments

The final accepted and deployed revision records structured before-and-after recommendations and checks the same incident input and normalized observations, one governed-memory intervention, and durable correction lineage before showing the controlled scale_workers → throttle_retries comparison. Generated procedural lessons remain audit-visible but excluded from strict retrieval until exact bound review; rejection preserves the candidate and reason without creating an active successor. Every visible claim is backed by persisted memory reads, decisions, fingerprints, human verdicts, and correction lineage. Stale memory is invalidated rather than deleted, retaining application-valid and transaction-time history. Public evidence reads are credential-free; mutation controls require the operator role. Durable queue delivery, leasing, reclaim, idempotency, dead-letter handling, and realtime replay are covered by the accepted exact-main path.

What we learned

Putting vector memory and transactional decision state in one database makes provenance much easier than coordinating a separate vector store. The useful unit is not merely a retrieved text chunk; it is a versioned belief connected to the run, evidence, recorded model recommendation, human decision, and correction that followed.

Limitations and next steps

This is a controlled incident scenario, not proof that Hindsight detects every poisoned memory or improves learning across an incident distribution. It does not claim service recovery, customer adoption, regional failover, or production-scale capacity. A direct diagnostic exercised 75,000 vectors across 15 tenants; the larger 100,000-vector, 20-tenant target was not established within its bounded attempt.

Next, we would evaluate recommendation quality across a labelled incident set, exercise the hosted tenant lifecycle drill, measure queue recovery and vector performance under representative load, and expand the recommendation catalogue only where each new recommendation has an equally strict evidence contract.

Built With

Share this project:

Updates