RecallGuard

AI agents remember everything, but who decides what they're allowed to know? RecallGuard: governed, auditable, human-approved memory for enterprise agents.

Comment

Inspiration

The moment that crystallised RecallGuard happened during a conversation about enterprise AI adoption. A CISO asked a simple question: "If our AI agents are reading customer data and writing what they learn into shared memory, who controls what they're allowed to remember?"

There was no good answer. Tools like Mem0, Zep, and LangMem are optimised for recall speed; they treat agent memory as a write-once, read-many cache where agents freely share what they learn. In a startup, that's fine. In an enterprise environment handling patient records, legal documents, financial data, or acquisition targets, it's a compliance disaster waiting to happen.

The gap is real: a hallucinated fact written by one agent becomes context for every downstream decision. A memory containing unredacted PII lands in an uncontrolled store, invisible to auditors. A regulator asks, "What did the agent know when it made this decision?" and there's no answer.

We built RecallGuard because enterprise AI doesn't just need faster memory; it needs governed memory.

What it does

RecallGuard is an enterprise-grade governed memory control plane for AI agents. The core guarantee is simple: an agent can only use knowledge that a human has explicitly approved, and every action is permanently recorded.

The governance loop:

  1. An agent writes a memory via REST API or SDK, returns in under 300ms; the agent is never blocked
  2. Amazon Comprehend scans for PII automatically; Bedrock Titan Embeddings v2 generates a 1024-dimensional vector
  3. The memory sits in pending, invisible to all agent retrieval until reviewed
  4. A human reviewer in the dashboard approves, rejects, edits, or marks it sensitive
  5. Approval triggers an atomic transaction: all four writes (memory status + review record + version history + audit log) commit together, or none do
  6. Approved memories become available for semantic kNN search; rejected ones are permanently excluded

Beyond the core loop, RecallGuard ships as a complete enterprise platform:

  • Team management and RBAC — four roles (org_admin, workspace_admin, reviewer, viewer) enforced at the API layer on every route, not just hidden in the UI
  • HMAC-signed invite flow — org admins invite team members; invitees land on /join/[token] showing their org and role before authenticating
  • Smart ingestion — paste raw text; Bedrock Claude Haiku extracts structured facts with confidence scores; reviewers approve or reject per item
  • Webhooks with Slack Block Kit — governance events fan out to registered endpoints; Slack URLs auto-receive formatted cards
  • GDPR Article 17 erasure — one workflow replaces all memory content for a given agent while preserving the audit trail
  • S3 audit export — NDJSON with SHA-256 hash, Object Lock (Governance mode, 365-day retention)
  • Python and TypeScript SDKs — three-line integration for any AI agent

How we built it

Three runtime boundaries, zero long-lived credentials.

Boundary Runtime Auth
Vercel (Next.js App Router) Node.js serverless, lhr1 London OIDC token exchange → AWS STS
Lambda embedding worker Node.js 20, SQS trigger IAM execution role
Demo agents Python 3.11 RecallGuard Bearer token

Aurora DSQL was the architectural keystone. The governance guarantee — all four writes or none — requires true ACID transactions across a serverless database. DSQL gave us that at scale with no infrastructure to manage. It forced some discipline: no foreign keys, no TEXT[] arrays (use JSONB), CREATE INDEX needs ASYNC, and auth tokens expire in ~15 minutes so the connection pool password must be an async function generating a fresh IAM token per connection.

The async pipeline decouples write latency from heavy processing. POST /api/v1/memories inserts the record and emits an SQS message in under 300ms. Lambda picks it up asynchronously: Comprehend PII detection → Bedrock Titan Embeddings v2 (1024-dimensional vectors) → OpenSearch Serverless kNN index → DSQL status update. The agent never waits for any of this.

OpenSearch Serverless (AOSS) taught us several lessons the hard way. AOSS auto-generates document IDs — you cannot specify your own — so memory_id lives in the document body, not the _id field. kNN queries cannot nest inside bool.must; they must be top-level. And the status filter (approved-only) is done in DSQL during record hydration, not in OpenSearch, because DSQL is always the authoritative source of truth.

The invite system uses HMAC-SHA256 signed tokens — base64url(JSON.stringify({email, orgId, orgName, role, exp})) + . + HMAC — with a 7-day TTL. No invite table needed. The token is self-contained, verified server-side, and the /join/[token] page shows the invitee exactly what they're joining before they authenticate. After OTP verification, provision detects the pre-created users row and routes directly to /dashboard, bypassing the onboarding wizard entirely.

Challenges we ran into

Aurora DSQL quirks. No Terraform provider exists yet — the cluster must be provisioned manually via the AWS Console. No foreign keys. No array types. IAM token auth with ~15-minute expiry meant warm Lambda and Vercel instances were hitting expired tokens mid-flight until we made the pool password an async function that generates a fresh token per new connection. And CREATE INDEX silently fails without the ASYNC keyword.

OpenSearch Serverless IAM. Both the Vercel OIDC IAM role and the Lambda execution role must be listed as principals in the AOSS data access policy. Missing either one produces a 403 that looks like a network issue, not a permissions issue. We spent meaningful time on this.

Bedrock in eu-west-2. Anthropic models require an AWS Marketplace subscription on first invocation in a region — and that subscription must be made by the correct account. Claude Haiku 4.5 needs the EU cross-region inference profile (eu.anthropic.claude-haiku-4-5-20251001-v1:0). Titan Embeddings v2 uses a direct model ID with no Marketplace step. These are not documented together anywhere.

Webhook delivery architecture. EventBridge is provisioned and wired up, but API Destinations — the piece that makes EventBridge actually deliver HTTP webhooks — require a separate Connection resource and rule configuration that Terraform doesn't express cleanly alongside custom event patterns. We built direct HTTP fan-out from DSQL webhook registrations instead, which turned out to be faster and simpler: read the registered endpoints, POST to each, sign with HMAC-SHA256, detect Slack URLs and format Block Kit. EventBridge now serves as an audit/event bus for future rule-based automation.

The invite routing bug. Invited users were being asked to name their organisation — because the provision route returned { provisioned: true } regardless of whether the user was joining an existing org or creating a new one, and the login page used provisioned: true to decide to send people to /onboarding. The fix was a single isNewOrg flag in the provision response — but finding the bug required thinking through the full session lifecycle from scratch.

Accomplishments that we're proud of

The governance transaction is a real database guarantee, not an application promise. Four writes — UPDATE memories, INSERT INTO reviews, INSERT INTO memory_versions, INSERT INTO audit_log — inside a single BEGIN/COMMIT on Aurora DSQL. If any write fails, all four roll back. You cannot have an approved memory without an audit trail entry. That's not defensive coding; it's a constraint enforced by the database engine.

The audit log is genuinely append-only. There is not a single UPDATE or DELETE statement against audit_log anywhere in the codebase. GDPR erasure replaces memory content — the memories table — but audit rows are permanent. When a regulator asks "who approved memory X, when, and what did the original content say?", the answer is always there.

Role enforcement lives at the API layer. All 29 API routes call requireRole() before touching any data. A viewer cannot call the review endpoint even with direct curl and a valid session cookie. The sidebar hiding is cosmetic; the API is the real boundary.

Zero long-lived AWS credentials on Vercel. Every AWS call from Vercel goes through OIDC token exchange (VERCEL_OIDC_TOKEN → fromWebToken() → short-lived credentials). No AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY stored anywhere on the platform.

The demo agents use the real Anthropic Claude API. Not simulated. ResearchAgent calls Claude Opus 4.5 to extract structured facts from a firmware incident report and writes them to RecallGuard. AssistantAgent calls semantic search and injects approved memories into Claude context. The WiFi hypothesis — rejected by a human reviewer — is completely absent from retrieval even when searched for by exact content.

What we learned

Serverless ACID is underrated. Aurora DSQL made the four-write governance transaction trivially correct. We didn't need to design compensating transactions, saga patterns, or eventual consistency workarounds. For workloads where correctness is the product — and in governance infrastructure, it absolutely is — a database that gives you real transactions is worth every quirk of its API surface.

Async architecture changes the product conversation. Returning in under 300ms with status: pending meant we could make the agent experience completely non-blocking while still requiring human review before retrieval. If we had made the review synchronous in the write path, agents would have needed to poll or wait. The SQS → Lambda pattern freed us from that tradeoff entirely.

Security boundaries need to be at the API, not the UI. We initially built the role system as sidebar filtering — certain pages just don't render for certain roles. A security review reminded us that hiding navigation doesn't protect APIs. Adding requireRole() to all 29 routes took a few hours but fundamentally changed what the product guarantees. The UI hiding is now just a UX convenience on top of real enforcement.

Self-contained signed tokens beat database tables for short-lived state. Invite tokens carry {email, orgId, orgName, role, exp} in HMAC-signed base64url. No invite table, no cleanup job, no expiry cron. The token is verified server-side in one function call. The same pattern applies to password reset links, export download tokens, and webhook secrets.

AWS service composition has hidden dependencies. Every service we used had at least one undocumented integration constraint: AOSS requires both principals in the data access policy; Bedrock Anthropic models require Marketplace subscription; Lambda + DSQL needs async IAM token refresh; EventBridge API Destinations need a separate Connection resource. These took longer to debug than the actual feature code.

What's next for RecallGuard

Memory confidence decay. Semantic facts become stale. A memory written six months ago about a system's architecture might be actively misleading today. We want to model confidence decay over time — memories with high source_age and low retrieval frequency gradually move toward expired with a prompt for human re-review, not automatic deletion.

Cross-agent provenance graphs. When Agent B retrieves a memory that Agent A wrote, that's a lineage edge. Building a provenance graph over the audit log would let compliance teams trace exactly which approved facts influenced which downstream agent decisions — the answer to the question regulators will actually ask.

Policy-as-code. Today, access policies are set per-agent per-space via the dashboard. We want to support declarative policy files — recallguard.policy.yaml — that can be version-controlled and applied via CI/CD, so governance policy evolves alongside the codebase that consumes it.

MCP server for agent frameworks. A Model Context Protocol server that exposes RecallGuard's approved memory store as a native context source for Claude, LangChain, LlamaIndex, and Vercel AI SDK agents — so any MCP-compatible agent gets governed retrieval without changing application code.

Differential privacy for aggregate analytics. The analytics dashboard today shows raw counts. For sensitive deployments where even aggregate memory statistics could leak information, we want to add differential privacy noise to the reporting layer — useful for healthcare and financial services use cases.

Federated review queues. Large organisations will want domain-specific reviewers: the security team reviews memories tagged infrastructure, the clinical team reviews memories tagged patient-data. A routing layer that assigns pending memories to the right queue based on tags, workspace, or PII entity type — with SLA tracking per queue.

Built With

Share this project:

Updates

Submission history