HindSight

Judge the decision, not the hindsight. HindSight reconstructs what an AI agent could know at decision time, so a wrong outcome stops automatically meaning a faulty agent.

Comment

Inspiration

Every post-mortem on an AI agent runs into the same trap. Something went wrong, you open the database, and the data tells you what is true now. So the agent looks incompetent — when quite often it was reasoning correctly over the only record that existed at the time.

I wanted to build the thing that can tell those two situations apart, because they demand opposite responses. A faulty agent needs to be fixed. An agent that was starved of data needs a fixed pipeline, and blaming the model instead sends the whole team chasing the wrong problem.

The reference case is deliberately small enough to verify by hand: a telecom call billed at €0.25/minute, a tariff corrected retroactively to €0.15, and a customer overcharged by one euro. The correction had legally taken effect before the call was billed — it just did not become queryable in the database for another 48 hours.

What it does

HindSight reconstructs the moment of a decision and issues a verdict on it.

It replays what was actually true, what the agent could query at that instant, what evidence was available, what was retrieved, what was shown to the model, and what was finally used. The verdict here is wrong_not_knowable: the outcome was financially wrong, and the agent is not at fault.

The audit itself is fully deterministic. Bedrock is used only for bounded advisory English, and never touches a verdict or a refund amount. The remediation is idempotent: one corrected invoice, one refund, the dispute closed and one ingestion incident opened, in a single transaction — replay it and nothing duplicates.

A second dispute on the same route then retrieves the earlier investigation through CockroachDB's Distributed Vector Index, loads four verification steps, and changes the recommended plan. Memory advises; it never decides.

How I built it

Facts are stored bi-temporally in CockroachDB. Every assertion carries a business validity window (valid_from / valid_until) and a system knowledge window (recorded_at / superseded_at). Corrections insert a new immutable version instead of overwriting, so "what was knowable at time T" is a query rather than an archaeology project.

On top of that sit three bounded agents sharing one correlation ID, a client-side Bedrock Converse tool-use loop exposing a single case-scoped read-only tool, and an optional CockroachDB Cloud Managed MCP transport that serves that tool through one fixed select_query — the model never generates SQL.

The web service runs on ECS Express Mode: a Fargate task behind an HTTPS load balancer and AWS WAF, with the image pinned by sha256 digest and credentials injected from Secrets Manager. Every request emits structured spans under one correlation ID, from the HTTP call through the Bedrock and MCP calls down to the CockroachDB writes.

Challenges I ran into

TLS failed in the container and nowhere else. psycopg[binary] ships its own bundled OpenSSL, compiled with a default certificate path that does not match Debian's. The CA bundle was present in the image and libpq simply never looked at it, so sslmode=verify-full failed with a misleading "root certificate file does not exist". Fixing it meant pointing sslrootcert at an explicit path. I caught this by running the exact production image against the real cluster before touching AWS — it would otherwise have surfaced as a CloudFormation rollback.

The startup health check gave false confidence. The readiness gate validates the connection pool and the rate-limit tables. It does not touch the 16 business tables. My runtime role had grants on only 3 of 19 tables, so the service would have started green, served a healthy dashboard, and failed at the first audit. A health check that does not exercise the business path is a health check that lies.

A concurrency lease that could expire mid-call. The provider concurrency lease was sized against an assumed 15-second MCP budget. But the MCP client applied that timeout to each phase separately — initialize, list tools, call tool — with no deadline over the whole exchange, so the real worst case blew past the lease TTL. An expired lease silently stops capping concurrency. The fix was a global deadline, plus deriving the lease TTL from the provider timeouts in code so the two can never drift apart again.

Saturation that looked like a crash. psycopg_pool raises PoolTimeout when the pool is exhausted and TooManyRequests when the waiting queue is full — two unrelated exception types for the same condition. Both now translate into 503 database_capacity_unavailable, so pool saturation shows up as measurable shed load instead of hiding inside the generic 500 rate.

Rate limits that protected against the wrong thing. Provider budgets are token buckets with no rollback. Consuming budget before checking execution capacity meant a request rejected for lack of a slot still permanently burned spend. Taking the concurrency lease first, and releasing it if the budget check then fails, closes that.

Accomplishments I'm proud of

The whole path is traceable: one correlation ID follows a request from HTTP through the agent runs, the Bedrock calls and the MCP query, down to the rows in CockroachDB.

Deploying the service beside its database cut a full audit from 15.0 s to 8.5 s — the same image, the same cluster, network distance alone. That is a measured justification for an architectural choice rather than a preference.

And the README claims nothing that has not been measured. No percentiles, no throughput, no user counts.

What I learned

Derive constraints, don't restate them. The lease TTL is now computed from the provider timeouts; raising a timeout moves the TTL automatically. Every constant I kept in two places eventually disagreed with itself.

Test the real artifact against the real dependency before spending money. A local run of the production image against the production database caught three separate blockers in under an hour, each of which would have cost a failed deployment.

And stating limits explicitly is not a weakness. "One task, single observations, no throughput claim" is a stronger sentence than a percentile I cannot reproduce.

What's next

A durable job queue so a long synchronous audit survives task replacement with a strict SLA. A load run under the authorized procedure to turn single observations into real p50/p95/p99. And distinguishing "the pool is saturated" from "the database is unreachable", which today share a status code.

Built With

Share this project:

Updates

Submission history