ASTRA

Mention something to your car on the drive home, and your home assistant already knows it. ASTRA is one AI companion with one memory across car, office, and home, live on CockroachDB.

Comment

Inspiration

Many people now talk to separate assistants in their car, at the office, and at home. Every one of them starts from zero. On the drive home you mention the left rear tire keeps losing pressure. Twenty minutes later you ask the speaker in your kitchen whether there is anything you should handle this weekend, and it has no idea what your car heard. You repeat yourself across devices all day, and the things you only said once quietly disappear. I built ASTRA because this is my daily life: I talk to assistants in three places, and every conversation dies where it happened.

What it does

In the car you say: "remember the left rear tire is losing pressure." At home that evening you ask: "anything I should deal with this weekend?" ASTRA answers, and names where it heard it: you mentioned this while you were in the car. Same identity, same memory, different interface.

ASTRA at home, proactively citing what it heard in the car

Try it live: https://astra.hcytlog.com/ (each visitor gets an isolated memory space, no signup).

Under the hood, in user terms:

  • One memory, three types. Episodic events (what happened), semantic facts (what is true about you), and procedural preferences (how you like things done) live in one CockroachDB schema, so a fact learned anywhere is recallable everywhere, with provenance.
  • Memory that respects place. Every memory carries a privacy scope. Semantic facts follow you across contexts by default; episodic moments stay scoped to where they happened. Memories can expire and be soft-deleted, so forgetting is a feature.
  • Conversations that travel. Session state is stored per user, not per device: start a thread in the car, continue it at home.
  • Hybrid recall. Retrieval combines vector similarity with keyword search, so exact names and model numbers are not lost to embedding fuzz.
  • Real scenarios. 24 tools across the car, home, and office surfaces, covering 5 event families (calls, email, vehicle, home, calendar).

We compared six agent-memory systems (Letta, Mem0, Zep, LangMem, A-MEM, CarMem). All six attach memory to a single agent or a single application. In that survey, none supports cross-context recall with source attribution as a first-class primitive. That gap is the product.

How we built it

ASTRA architecture: three scenes, one gateway on EC2, Bedrock reasoning, one CockroachDB Cloud schema

The diagram above is the whole system: three scenes hit one Fastify + TypeScript gateway on EC2, reasoning runs on Amazon Bedrock (Gemma 4 31B via the Mantle endpoint), and every memory operation lands in one CockroachDB Cloud schema, all in us-east-1.

The design is one ledger, three memory types. Instead of three stores, all memory rows share a single schema with type, privacy scope, expiry, soft deletion, and typed links between related memories. Cross-device session state lives on the same cluster, so the database, not the device, is the source of identity. Retrieval is hybrid, Voyage AI embeddings plus keyword search, so exact names and model numbers survive embedding fuzz. Caddy and Cloudflare terminate HTTPS because browsers only grant microphone access over HTTPS, and cookie-scoped tenancy gives every visitor of the public demo an isolated memory space.

Challenges we ran into

  • The safe default was the wrong default. We started by scoping every memory to the context that created it, which felt privacy-correct until the core scenario failed: a fact learned in the car was invisible at home. The fix was to split defaults by memory type: semantic facts cross contexts, episodic moments stay where they happened. Privacy scope turned out to be a property of the kind of memory, decided in the schema.
  • Streaming died in production. Token streaming worked locally and broke once deployed behind the EC2, Caddy, and Cloudflare chain. The root cause was Caddy's response buffering: it held SSE chunks until the entire response finished, so the user saw nothing for 30 seconds and then a wall of text. Disabling buffering per-route and switching the SSE close detector from the request socket to the response socket fixed it.
  • A public demo is a shared machine. For a product about personal memory, visitors sharing one memory pool is disqualifying. Cookie-scoped multi-tenancy now gives each judge a private memory space with zero signup friction.
  • The model was reading our internal notes out loud. Memories that crossed scenes were tagged [handoff] in the prompt so the model would know to raise them proactively. It did raise them, and it also pasted the literal tag into replies, so users saw "[handoff] your wife's birthday is Saturday." The tag also carried no scene name, so the model had to guess where a memory came from, and often skipped saying it at all. We replaced it with the scene itself ([said in the car]) and forbade printing bracketed tags. After the change the model states provenance in its own words ("you mentioned this while you were in the car"), because the label now carries the scene instead of an internal codeword.

Accomplishments that we're proud of

  • Cross-context recall, measured: 14/15 recalled, 13/15 cited their source. Each run stores 5 memories across the driving and office scenes (episodic events, semantic facts, procedural preferences), switches to a third scene, and asks 5 recall questions. Across 3 runs on the live deployment: 14 of 15 answers recalled the right memory with detail, and 13 of 15 named the scene it came from. No miss was a fabrication: the one recall miss surfaced a different real memory about the car, and the attribution misses recalled the right fact but skipped the scene label. tests/recall-benchmark.sh reruns the whole thing against the public URL; raw per-query results are in the repo.

Benchmark: 14/15 recalled, 13/15 cited their source scene

  • The infrastructure is real, and the demo says so out loud. CockroachDB Cloud, Bedrock inference, Voyage embeddings, EC2, Cloudflare HTTPS: every one is a live managed service, no local containers. What we cannot own is the last mile into someone's actual car and house, so the demo ships a Data Sources panel that labels each feed: clock and geolocation are LIVE, weather/places/navigation are LIVE when you grant GPS, calendar is simulated against the real clock, and home and vehicle control are MOCK behind a HomeKit-shaped and vehicle-SDK-shaped interface. Judges can see exactly where the real system ends.
  • A gap we could not find filled elsewhere. Six agent-memory systems surveyed (Letta, Mem0, Zep, LangMem, A-MEM, CarMem). All six bind memory to a single app. In that comparison, none treats cross-context recall with provenance as a core primitive.
  • 3 memory types, 24 tools, 3 scenario surfaces, 5 event families, all on one CockroachDB schema.

What we learned

Recall quality is a retrieval problem. Trust is a scoping problem. An AI that follows you across contexts has to solve both, and the scoping has to live in the schema, not in the prompt, because prompts change with every model swap. Everything hard about this project came down to one design choice: making privacy scope a column in CockroachDB, not a rule in the system prompt.

What's next for ASTRA

Production hardening first, named honestly because we know exactly where the gaps are:

  • Idempotent memory writes: a transaction ledger so a retried event cannot double-write a memory.
  • Persisted approval state and an audit table for actions the agent takes on your behalf.
  • Least-privilege database roles: the demo currently runs on a broader-privileged account than production should.

Then measurement and reach:

  • Expand the recall benchmark: 15 queries over 3 runs is a starting point, not a claim about the general case. Next is a larger labeled memory set (50+ items, more scenes) with automated scoring, plus the adversarial cases we do not test yet: conflicting memories, expired items, and privacy-scoped items that must not surface.
  • More surfaces on the same memory: watch and TV clients are thin once the memory layer is the product.

Built With

Share this project:

Updates