Continuum
An autonomous incident-response agent that resumes the exact step it was killed on — because its memory lives in CockroachDB, not in the process.
CockroachDB × AWS Hackathon 2026 — Build with Agentic Memory
What Is This?
Most "agent memory" demos store chat history. Continuum stores something that matters under pressure: which remediation step is executing right now, which alert correlates with which past incident, and the exact state of recovery the instant before something crashes.
Every state transition is committed to CockroachDB before and after it happens. Kill the process mid-step — no graceful shutdown, no checkpoint call — and the next cold invocation reads the durable state, sees a step frozen in executing, and resumes that exact step. No lost context, no duplicated work, no human re-input.
All incident and alert data is synthetic. No real production systems, credentials, or customer data.
The Problem
The conditions that cause production incidents — resource exhaustion, node failure, deploy rollbacks, autoscaling churn — are exactly the conditions that kill the agent responding to them. An agent holding its working state in process memory doesn't degrade gracefully when that happens. It stops, and a human restarts the incident from zero, without knowing which actions already ran.
That makes "did this step already execute?" the most expensive question in an incident: re-running a remediation action can be worse than not running it at all.
The agent's execution environment is allowed to die mid-incident. Its memory is not.
Continuum treats that as a design constraint rather than an edge case. The recovery path is not error handling bolted onto a happy path — it is the only path, exercised on every single invocation.
How It Works
- A synthetic alert fires (latency spike, error-rate breach, connection saturation)
- The Orchestrator (AWS Lambda) does the same thing whether its execution environment is brand new or reused — its first action, always, is a CockroachDB recovery read for open incident state matching this alert
- The Correlation Agent embeds the alert via Amazon Bedrock (Titan v2, 1024-dim) and queries CockroachDB's C-SPANN vector index for semantically similar past incidents — structured filters and semantic ranking in one SQL round trip
- The Remediation Agent reasons over the matched precedent (Claude on Bedrock) and proposes the next step
- The Memory Agent — the only module allowed to write state — commits each step in explicit
SERIALIZABLEtransactions: the proposed action andexecutingstatus together (a forward step is claimed exactly once,ON CONFLICT DO NOTHING), thenexecuted, withresolvedcommitted atomically alongside the final step chaos_kill.pyhard-kills the process mid-execution; the step stays durablyexecutingin CockroachDB — the fingerprint the next invocation resumes from- The Query Agent answers live questions through the CockroachDB Cloud Managed MCP Server — "show me all open incidents and their current remediation step" — from
GET /api/v1/incidents/openand the Gradio UI's "Ask via MCP" button, not just from a human typing into an IDE
Architecture
Click to enlarge (opens the full-resolution SVG — scales without pixelation): light / dark · Source: architecture-diagram.mmd — rendered to brand-themed SVG/PNG (dark + light, plus 16:9 video cards) via mermaid-cli; see assets/architecture/README.md for the regenerate command.
In short: one invocation = one remediation step. The recovery read happens before any reasoning, every step commits in two SERIALIZABLE transactions with the execution window between them, and a forward step is claimed exactly once. The red path is the whole point — chaos_kill.py severs the process mid-step, and nothing about the recovery depends on that process ever coming back.
The Recovery Pipeline
The component diagram shows what talks to what. This shows what survives — two cold Lambda invocations, no shared memory between them, handing off entirely through durable CockroachDB state:
Click to enlarge: light / dark · Source: recovery-sequence.mmd
Step 2 is the one that matters. The recovery read is the first branch in orchestrator.py, before any new reasoning happens — not an error handler, not a retry wrapper. That is what separates Continuum from an agent that also happens to log to a database.
Deep dive →
docs/ARCHITECTURE.md— the dual memory model, the step-by-step recovery walkthrough, the vector-index DDL, and a typical-agent vs Continuum comparison.
Architecture Decision Records
Ten decisions documented (001–010), all accepted and implemented — see docs/adr/ for full rationale.
| ADR | Decision |
|---|---|
| 001 | Dual transactional + vector memory in one CockroachDB store — no separate vector DB to drift |
| 002 | Stateless Lambda, no provisioned concurrency — every invocation must recover cold |
| 003 | MCP Server in read-only mode as the live query interface |
| 004 | ccloud CLI evaluated, then cut — 2 tools done well beats 3 done thin |
| 005 | Synthetic incident corpus only — no real infra, ever |
| 006 | Explicit scope cuts, documented instead of hidden |
| 007 | eu-central-1 deployment region, kept in sync across config/template/ADR |
| 008 | Bedrock calls target their own BEDROCK_REGION setting rather than reusing AWS_REGION, so Bedrock can move without redeploying the Lambda — introduced when a dynamic account-level quota clamp probed as ~0 across all regions and models (lifted 2026-08-01); the default is back to eu-central-1 alongside the Lambda and cluster (addendum 3), and the app degrades to deterministic fallbacks either way |
| 009 | Each step runs in two explicit SERIALIZABLE transactions with a forward-step claim (ON CONFLICT DO NOTHING) for exactly-once; correlation/Bedrock is best-effort, off the recovery critical path |
| 010 | The orchestrator deploys from CI on a v*.*.* tag — via GitHub OIDC, never stored AWS keys — so the deployed function and the newest tag cannot drift. Tags only, never pushes to main: redeploying during ordinary work would swap the code out from under a demo recording |
CockroachDB Tools Used — and what the agent actually does with them
Two tools, both load-bearing in the running application (see ADR 004's resolution on why that's two done well rather than three done thin):
- Distributed Vector Indexing —
incident_embeddings.embedding VECTOR(1024)with a C-SPANN index prefixed byservice, so ANN search partitions per-service. The Correlation Agent's live query filters by structured columns and ranks by<->distance in one round trip. Seeinfra/schema.sql. - CockroachDB Cloud Managed MCP Server — read-only mode;
agents/query_agent.pyis a real MCP client (officialmcpSDK, streamable HTTP) that the app itself calls fromGET /api/v1/incidents/openand the Gradio UI's "Ask via MCP" button — not only a development convenience. The server's audit log doubles as a trail of what the agent looked at.
AWS Services Used
- AWS Lambda — orchestrator execution on the
python3.14runtime; deliberately no provisioned concurrency, so every invocation proves state comes from CockroachDB, not warm process memory (ADR 002) - Amazon Bedrock — Titan Text Embeddings V2 for alert→vector; Claude Sonnet 4.5 for remediation reasoning over matched precedent (with a deterministic precedent-replay fallback so the control flow demos even when throttled)
- AWS SAM — infrastructure as code (
infra/template.yaml); the absence ofProvisionedConcurrencyConfigis a reviewable artifact rather than a console setting - AWS IAM — least privilege: application credentials are scoped to Bedrock model invocation only and cannot list, create, or delete AWS resources
Judging-criteria mapping and full submission narrative: submission/DEVPOST.md
Tech Stack
| Layer | Technology | Role |
|---|---|---|
| Memory | The durable record. Transactional incident state and vector embeddings in one store — no second database to drift (ADR 001) | |
| Durability | Two explicit transactions per step — executing committed before the execution window, executed after. A kill lands with executing durable (ADR 009) |
|
| Correlation | service-prefixed C-SPANN index; structured filter + <-> ANN ranking in one round trip (infra/schema.sql) |
|
| Live Queries | The app itself is the MCP client, not just a developer's IDE — GET /api/v1/incidents/open and the UI's "Ask via MCP" (ADR 003) |
|
| Agent Pattern | Orchestrator · Correlation · Remediation · Memory · Query. memory_agent.py is the only module permitted to write state |
|
| Embeddings | Alert → 1024-dim vector, matching the VECTOR(1024) schema |
|
| Reasoning | Next-step proposal over matched precedent, with deterministic precedent-replay fallback so the flow demos even when throttled | |
| Compute | Stateless orchestrator, deliberately no provisioned concurrency — every invocation proves cold recovery (ADR 002, infra/template.yaml) |
|
| Backend | Versioned gateway (/api/v1) around the orchestrator; psycopg 3 because psycopg2 has no 3.14 wheels |
|
| Demo UI | Live incident console with recovery-timeline replay, the recalled precedent per step, and the committed failure evidence — reading straight from CockroachDB, in the viewer's own light or dark theme. Every card carries provenance badges naming the CockroachDB and AWS features that produced it (⟲ resumed after kill, ⌖ recalled #N of M, the embedding and reasoning models, the runtime), and a legend names the column each one is read from — so the tool claims are checkable against the database, not taken on trust |
|
| Observability | Structured event logging across every agent — no bare print |
|
| Quality | Lint → format → types → 88 unit + 9 integration tests → 100% coverage against a 90% gate → Codecov |
Live Demo
| App | https://huggingface.co/spaces/iarjunganesh/continuum (deploys on push to main) |
| Orchestrator | Live on AWS Lambda — continuum-orchestrator, eu-central-1 (stack continuum), deployed from CI on a version tag (ADR 010). No provisioned concurrency: cold start 1806 ms init, 130 MB / 512 MB (sampled 2026-08-08 on the v0.9.5 build; every tag since has changed no application code and sits inside the same 1578–1806 ms spread). Warm environments are reused between back-to-back invocations, and it doesn't matter — the orchestrator re-reads CockroachDB first either way, so the guarantee never rests on the container being new. docs/DEPLOY.md is the authority on what is currently live |
| Demo Video | https://youtu.be/LwD8__sKqa0 — 2:55, 1920×1080/30, captions included. The kill and the resume are one continuous take, no cut between them. Script: submission/DEMO_SCRIPT.md |
| Try It Now | make chaos-demo — kill the agent mid-incident, watch it resume from CockroachDB |
Submission checklist: submission/SUBMISSION.md · Judging alignment + project story: submission/DEVPOST.md · Cost model: submission/COSTS.md
Run it yourself — the interactive notebook
notebooks/DEMO_RUNBOOK.ipynb — a self-contained walkthrough of the kill-and-recover sequence. The recovery guarantee is easy to assert and hard to believe without watching it, so step through it yourself rather than taking the README's word.
| Section | Needs a local API? | What you'll see |
|---|---|---|
| 1 — Fire a synthetic alert | No | Recovery read runs before any reasoning; correlation finds a precedent |
| 2 — Read state back over MCP | No | The app calling the Managed MCP Server's read-only SQL tool, live (ADR 003) |
| 3 — Advance one step | No | Two SERIALIZABLE commits per step with the execution window between them |
| 4 — The kill | Yes | A real SIGKILL landing mid-step — no graceful shutdown, no checkpoint call |
| 5 — State outlived the process | Yes | The row sitting in executing with nothing alive to own it — the whole thesis |
| 6 — The recovery | Yes | That exact step re-executed, not skipped and not duplicated |
pip install -r requirements.txt jupyter
make run-api # in a separate terminal
jupyter lab notebooks/DEMO_RUNBOOK.ipynb
Setup notes and conventions: notebooks/README.md.
Captured Evidence
Judge-facing evidence — real runs with raw CockroachDB snapshots, structured logs, and provenance manifests recording the exact commit, cluster and models used. Index: assets/README.md.
A real kill-and-recover run is captured and committed: assets/chaos-run/local-a2bb201d/ — a live orchestrator hard-killed mid-step (real SIGKILL, pid 38600, no graceful shutdown), read back out of the database at three phases and photographed in the providers' own consoles at each one:
| Phase | What CockroachDB said |
|---|---|
01-before-kill |
step 0 executing, process alive |
02-frozen |
process dead — the step still executing, with nothing alive to own it |
03-after-resume |
incident resolved, all 3 steps executed exactly once, none duplicated |
Captured by make chaos-capture, which performs the kill and records it, and marks the folder FAIL rather than emitting evidence if the kill misses the execution window or a step runs twice. correlation_source and reasoning_source both read bedrock, so the live AWS path is provable from the rows rather than assumed. The 02-frozen row is the one that cannot be staged after the fact — once a run resolves the console reads resolved — which is why --pause exists and why that frame is in local-a2bb201d/screenshots/.
And the same three phases on the deployed function, with AWS delivering the kill: assets/chaos-run/lambda-0b99a950/. make chaos-capture-lambda lowers the function's own timeout below its step-execution window, so Lambda terminates the invocation mid-step — no signal the runtime can catch, no cleanup, no checkpoint — and the resume is a second cold invocation of the same function. Step 0 froze executing with nothing alive to own it, resumed at that exact index, and the incident resolved with 3 steps executed and 0 duplicated, every durable step recording runtime: lambda from the function's own environment. The folder carries the function's own CloudWatch log for the window, which reads in order: INIT_START on python:3.14 → step_checkpoint_start → REPORT … Status: timeout → a second INIT_START → recovered_incident_state with last_step_status: executing. That is the entire thesis, written by AWS rather than by this project.
Alongside it: assets/resilience-run/ (kill storms, AWS-initiated Lambda timeouts, exactly-once under 50-way concurrency, vector search to 10,000 vectors), assets/deploy-restart-run/ (the function's code replaced under an open incident), and assets/provider-evidence/ — the same facts in Cockroach Labs', Hugging Face's and AWS's own consoles, screens this project cannot fake.
The providers, saying it themselves
Four of those frames, one per claim. Every number is rendered by a console this project does not
control — click any image for the full-resolution capture, and see
assets/provider-evidence/README.md for what each one does
and does not establish.
Console screenshots are captured for both runs (2026-08-09) —
local-a2bb201d/screenshots/andlambda-0b99a950/screenshots/, 7 frames each, shot live inside the--pausewindow where the frozen row exists at all. Recovery on the deployed function is evidenced six further ways beyond its own capture: the deploy-restart drill, the AWS-timeout suite where AWS performs the kill, cold-invocation latencies indocs/BENCHMARKS.md, durable steps recordingruntime: lambdaalongside the model ids that produced them, and — since 2026-08-08 — the function's own CloudWatch logs in both forms: as text (assets/provider-evidence/13.lambda-recovery-reads.txt) and as a console frame (assets/provider-evidence/11.lambda-log-stream-recovery.png), both showingrecovered_incident_statewithlast_step_indexacross cold invocations of onecorrelation_id. Open gaps are tracked insubmission/SUBMISSION.md.
Quick Start
# 1. Clone
git clone https://github.com/iarjunganesh/continuum.git
cd continuum
# 2. Configure (CockroachDB Cloud free tier + AWS credentials)
cp .env.example .env # fill in COCKROACH_DATABASE_URL + AWS keys
# 3. Install (requires Python 3.14)
make install
# 4. Apply schema + seed synthetic incident history (with embeddings)
make migrate
make seed-data
# 5. Run the API + demo UI
make run-api
make run-ui
# 6. The resilience demo — kills the agent mid-step, proves recovery
make chaos-demo
On Windows — PowerShell 7+
Windows has no make, so every step has a direct equivalent. The Makefile stays the source of
truth for what each step does; this is the same work, typed differently.
# 1. Clone
git clone https://github.com/iarjunganesh/continuum.git
cd continuum
# 2. Configure
Copy-Item .env.example .env # fill in COCKROACH_DATABASE_URL + AWS keys
# 3. Install (requires Python 3.14)
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
# 4. Apply schema + seed synthetic incident history
.\scripts\migrate_and_seed.ps1 # add -Offline for deterministic vectors, no AWS
# -SkipSeed for schema only, -Count N to resize
# 5. Run the API + demo UI (separate terminals)
python -m uvicorn api.main:app --port 8000
python ui/app.py
# 6. The resilience demo — kills the agent mid-step, proves recovery
.\scripts\chaos_demo.ps1
Every other make target is a one-line recipe you can read straight out of the Makefile — most
are a single python scripts/….py. The ones used most often:
make target |
PowerShell |
|---|---|
make probe-bedrock |
python scripts/probe_bedrock.py |
make check-drift |
python scripts/check_drift.py |
make chaos-capture |
python scripts/chaos_capture.py (add --pause to hold for screenshots) |
make chaos-capture-lambda |
python scripts/chaos_capture.py --via-lambda --profile continuum-admin |
make clean-clone-check |
python scripts/clean_clone_check.py |
make test |
pytest tests/unit tests/integration -v |
make lint |
ruff check . ; ruff format --check . |
make typecheck |
mypy agents/ api/ observability/ config.py |
make coverage |
pytest tests/unit tests/integration --cov=agents --cov=api --cov=observability --cov-report=term-missing |
Two targets are POSIX-only by design and have .ps1 equivalents rather than translations —
make chaos-demo (backgrounding and sleep) and make deploy ($$VAR expansion). Use
.\scripts\chaos_demo.ps1 and the steps in docs/DEPLOY.md.
The API is versioned under /api/v1 — e.g. GET /api/v1/health, POST /api/v1/alert — so the wire contract can evolve without breaking the Gradio UI or demo scripts.
Synthetic Demo Data
40 resolved historical incidents across 5 fictional services (checkout-api, auth-service, recommendation-engine, search-index, billing-worker), each seeded with its actual remediation path (e.g. drain_connection_pool → restart_connection_pool → verify_connections_healthy) — so when a live alert correlates with a precedent, the Remediation Agent has real steps to replay, not just a summary. Regenerate anytime:
python scripts/generate_synthetic_incidents.py --out data/synthetic/incidents_seed.jsonl --count 40
Seeding without Bedrock. Real Titan vectors for the 40 seed incidents are committed at data/synthetic/seed_embeddings.json, so honest semantic correlation needs no AWS call at seed time:
python scripts/seed_memory.py --file data/synthetic/incidents_seed.jsonl `
--from-fixture data/synthetic/seed_embeddings.json --replace-embeddings
--replace-embeddings overwrites vectors that already exist; without it the insert is ON CONFLICT DO NOTHING, which is right for topping up incidents and silently wrong when the point of the run is to replace the vectors. make seed-data embeds via a live Titan call per record instead; make seed-data-offline uses deterministic vectors that populate the table with no AWS dependency at all but are explicitly not semantically meaningful — measured precision@1 55% vs 98% for Titan on the same corpus, so prefer the fixture for anything a reader will interpret as correlation.
Project Structure
Omitted from this paste for length — GitHub renders the live tree, which cannot go stale:
browse the repository. The annotated version, checked against the real repo
on every commit, is in README.md.
Production & Quality
push → ruff lint → ruff format --check → mypy → Devpost mirror freshness
→ ephemeral single-node CockroachDB → schema apply
→ pytest (88 unit + 9 integration) → coverage (≥90% gate, 100% measured) → Codecov
push to main → auto-sync to Hugging Face Space (public demo)
tag v*.*.* → GitHub Release, notes pulled from CHANGELOG.md
→ deploy the orchestrator to AWS Lambda (OIDC, no stored keys)
→ assert CodeSha256 actually moved → smoke-test the deployed package
A tag deploys the function (ADR 010), so the deployed orchestrator and the newest tag cannot drift. Deliberately not on every push to main — that would redeploy the live function during ordinary work, including while the demo is being recorded against it.
See .github/workflows/ci.yml, .github/workflows/deploy.yml, .github/workflows/release.yml, and docs/DEPLOY.md.
The unit suite (88 tests, one file per agent/module, 100% measured coverage against a 90% CI gate) pins the properties the demo depends on: recovery read happens before any write, each step commits inside an explicit SERIALIZABLE transaction, interrupted steps are re-executed (never skipped, never duplicated), a forward step is claimed exactly once under concurrent invocations, and incidents resolve atomically with the final step. Three of them measure retrieval quality rather than control flow — recomputing the precision@1 figure this README quotes, so a claim about what the vector index means cannot go stale unnoticed (make precision-check).
tests/integration/test_recovery_e2e.py drives that same resume-and-exactly-once contract against the real schema on a real CockroachDB instance CI spins up — not just against mocks — and tests/integration/test_chaos_kill_e2e.py goes one step further: it spawns the orchestrator as a real subprocess and hard-kills it mid-step with scripts/chaos_kill.py (a real SIGKILL/TerminateProcess, no graceful shutdown), then asserts a cold restart resumes the interrupted step exactly once from CockroachDB. The same script drives the literal process-kill beat live in the demo.
Beyond tests: structlog JSON logging across every agent, secrets via environment only, least-privilege IAM, and documented scope cuts (ADR 006) rather than hidden ones. Security posture and known limitations: SECURITY.md. Cost model and guardrails: submission/COSTS.md.
Load & Resilience
What a remediation step actually costs, end to end: the CockroachDB legs (recovery read, both transaction commits, C-SPANN vector search, the full cold-resume path), the Bedrock legs (real Titan embedding), and the same work measured on the deployed Lambda rather than predicted from a workstation. Every run records which path actually executed — correlation_source / reasoning_source are counted, not assumed, so a throttled account can't quietly publish cheaper numbers under a Bedrock headline. make benchmark (add --with-bedrock --lambda-n N for the AWS legs; the default run needs no AWS). Full tables, methodology and caveats: docs/BENCHMARKS.md. Benchmarks run against make local-cluster, not the Cloud cluster serving the demo — docs/CLUSTER_OPS.md says which commands belong where, and why.
Speed is the less interesting half. The claim this project exists to make is about correctness when things go badly, so that is measured too — every number below counted from durable CockroachDB rows rather than from a log, against the live cluster and the deployed function. make resilience-bench; full method, sample sizes and caveats in docs/RESILIENCE.md, raw evidence under assets/resilience-run/.
| Failure mode | Result |
|---|---|
| Kill storm — 50 incidents interrupted mid-step | 50 resumed · 0 duplicated · 0 lost |
Real SIGKILL against a live process |
10 kills · 10 resumed · 0 duplicated |
| AWS Lambda timeout — AWS does the killing | 15 invocations · 15 killed by AWS · 15 resumed exactly once |
| Deploy mid-incident — the code replaced underneath an open step | sam deploy swapped the artifact · resumed on the new build, exactly once |
| Exactly-once under concurrent claimants | 100 trials, 5 levels up to 50-way · 0 violations |
| Concurrent agents | 10 / 50 / 100 · 53.7 completed/s · 0 failures |
| C-SPANN vector search, 100 → 10,000 vectors | 43 → 77 ms vs full scan 40 → 582 ms — 7.5× faster, against CockroachDB Cloud (a single-node local cluster narrows this — see RESILIENCE.md) |
The Lambda-timeout row is the one that can't be argued with: the process isn't killed by our own script but by AWS terminating the function, with no signal the runtime can catch and no opportunity to checkpoint. Every one of those recovered exactly once.
The deploy row is the only one that doesn't kill anything. It replaces the code under an incident already frozen in executing — a real sam build + sam deploy — and the cold invocation afterwards resumes that step on a build that did not exist when the step began. That's the failure an on-call engineer actually causes, by shipping a fix while an incident is open, and the durable row is the only thing bridging the two versions. make deploy-restart-drill; it compares CodeSha256 before and after and fails if the code didn't actually change, because a no-op deploy would otherwise pass while proving nothing. Evidence: assets/deploy-restart-run/.
tests/load/k6_smoke.js ramps concurrent users against /api/v1/health and the MCP-backed /api/v1/incidents/open, so the measurement covers the live MCP round trip rather than just FastAPI. It deliberately does not hammer POST /alert: that drives real state through the single write path, and exercising the forward-step claim outside controlled conditions would fabricate incidents rather than test them — exactly-once is proven in the integration suite instead.
winget install k6 # or: brew install k6
make load-test # local API
k6 run -e BASE_URL=https://<host> tests/load/k6_smoke.js # deployed
Roadmap (Post-Hackathon)
- Real alert-source integrations (PagerDuty/Opsgenie webhook ingestion) in place of the synthetic stream
- Multi-region incident correlation via
REGIONAL BY ROWincident tables - Contradiction/drift detection across recurring incident patterns
- Slack/Teams remediation-approval loop before a proposed step executes
Disclosure & Disclaimer
Built solo during the Submission Period (June 30 – August 18, 2026) with Claude Code as an AI coding assistant, per the hackathon's disclosure requirement. No pre-existing code was incorporated. All incident, alert, and remediation data is synthetic; Continuum is a technology demonstration, not a production incident-management tool, and is not affiliated with any company's real infrastructure.
Built by Arjun Ganesh for the CockroachDB × AWS Hackathon 2026.




Log in or sign up for Devpost to join the conversation.