Inspiration

Conversational and voice assistants today suffer from two fatal flaws: ephemeral amnesia and vector hallucination. Traditional RAG and memory layers rely on fuzzy semantic similarity, which cannot distinguish between an old plan and an updated decision—leading agents to confidently cite an outdated Friday deadline even after the team pushed it to Sunday. Furthermore, ambient devices blindly ingest background chatter, passwords, and sensitive PII into long-term context windows.

Inspired by research and open initiatives like Ambient Guard, Mental-Model-Drift, and MemoryAgentBench, we set out to build RecallOS: an evidence-grounded memory operating system for conversational and voice AI where every single memory is anchored to verbatim quotes, temporal changes are explicitly tracked, and privacy is enforced at the edge.

What it does

RecallOS converts continuous voice transcripts and team conversations into verifiable, structured memory vaults accessible by any AI agent via the Model Context Protocol (MCP):

Ambient Guard Privacy Firewall: Scans transcripts before extraction to redact API keys, credentials, credit card numbers, PII, and background TV/ambient noise.
Verbatim Quote Grounding: Memory nodes must match exact source transcript substrings. The agent can always answer why it believes a fact, citing the exact speaker, timestamp, and passage.
Temporal Supersession & Drift Radar: When plans change, RecallOS tracks the evolution rather than overwriting history. It surfaces cognitive divergence, milestone slippage, and unverified assumptions.
Calibrated Uncertainty: Classifies claims into confirmed, inferred, conflict, or unresolved—preventing agents from confusing absence of evidence with proof of absence.
17+ Protected MCP Tools & Voice Contract: Exposes tools (recall_get_briefing, recall_query_memories, recall_detect_drift, etc.) returning dual voice envelopes (spoken, grounded, nextStep, data) optimized for ambient screens like Echo Show.
10-Route Interactive Dashboard: Includes Daily Briefings, Temporal GraphRAG, Commitment Ledgers, Memory Consolidation pipelines, an Alexa+ Echo Show simulator, and Ring/AWS EventBridge device bridges.

How we built it

Frontend & Full-Stack SSR: Built with TanStack Start, React 19, and Tailwind CSS, utilizing Radix UI primitives and Recharts for deterministic fact visualization.
AI Gateway & Structured Extraction: Powered by the Vercel AI SDK with strict Zod schemas and custom substring validation algorithms ensuring exact quotation.
Multi-Tenant Vault & Persistence: Backed by Supabase and Drizzle ORM with strict Row-Level Security (RLS) guaranteeing user-level isolation.
MCP Server (/api/mcp): Fully typed JSON-RPC 2.0 server enforcing Bearer token authentication, input validation, and bounded agent loops (capped at 4 tool calls).
Smart Home Integrations: Ring Appstore authorization flow with AES-256-GCM encrypted tokens, Alexa Smart Home OAuth 2.0 endpoint handling, and HMAC-SHA256 validated webhook relays to AWS EventBridge via SigV4.
Testing: End-to-end test suite in Vitest covering redaction, grounding, drift detection, and security bounds.

Challenges we ran into

Sub-string Quote Provenance vs. LLM Paraphrasing: Large language models reflexively paraphrase source text. We had to build multi-pass extraction and deterministic string-alignment validators to reject hallucinations and ensure every quote exists verbatim in the raw conversation.
Supersession vs. Conflicting Beliefs: Differentiating between an actual agreed-upon schedule change (supersession) versus two teammates disagreeing on a detail (mental model divergence) required temporal sequence modeling and speaker attribution rather than standard vector proximity.
Designing for Voice & Screen Simultaneously: Voice assistants require ultra-concise spoken responses (1–2 punchy sentences), while multimodal displays and developers need deep metadata and citations. We designed a dual voice-envelope contract (spoken, grounded, nextStep, data) that satisfies both without compromise.
Securing Agentic MCP Endpoints: Preventing arbitrary agent turns from getting stuck in infinite retrieval loops or leaking private cross-tenant data required strict turn limits (max 4 steps) and caller-scoped RLS validation on every JSON-RPC dispatch.

Accomplishments that we're proud of

Zero Hallucinated Citations: Built a verifiable memory system where every claim points to an exact timestamped quote. If there's no match, it returns an explicit NO_MATCH_IN_CONNECTED_RECORDS outcome instead of fabricating an answer.
Comprehensive Privacy Pipeline: Implemented Ambient Guard, sanitizing API keys, passwords, and background noise before any text hits an LLM or database.
100% Green Core Test Suite: Wrote comprehensive unit and integration tests covering privacy redaction, temporal drift, unauthenticated MCP rejection, and bounded execution.
End-to-End Multimodal Experience: Delivered an interactive Alexa+ Echo Show simulator alongside 10 web dashboard routes, letting users inspect real-time JSON-RPC tool calls as voice interactions occur.

What we learned

Vector search is not memory: Pure embedding-based similarity retrieval is blind to time and sequence. Real conversational memory requires deterministic tracking of state transitions, supersession, and ownership.
Edge redaction is non-negotiable: Ambient conversational AI will fail user trust if privacy is handled as an afterthought; PII and credentials must be filtered before reaching prompts or vector databases.
Contract-first tool design: Designing structured tool envelopes with clear fallback wordings and certainty scores makes autonomous agents dramatically more predictable and easier to integrate across devices.

What's next for RecallOS

Versioned Memory Revisions: Transition from derive-on-read calculations to append-only consolidation event sourcing.
Official MCP SDK & SSE Streaming: Full conformance testing with the official Model Context Protocol TypeScript SDK, including SSE negotiation and OAuth 2.1 / PKCE metadata.
Cryptographic Evidence Hashing: Implement evidence SHA-256 fingerprinting, freshness decay scoring, and explicit INCONCLUSIVE / SOURCE_UNAVAILABLE states.
Durable Timezone-Aware Schedulers: Upgrade the proactive follow-up engine to deliver resilient, retry-safe reminders across multiple timezones.
MemoryAgentBench Regression Testing: Benchmark RecallOS against standard conversational memory benchmarks to continually refine drift detection precision.