Hearth — A Save Point for Agent Memory

Memories fade, resurface, and the agent chooses which ones to open.

Comment

Inspiration

Most memory systems treat agent context as a retrieval problem: store everything, search when asked. But human memory doesn't work that way — things slip away, come back uninvited, and some moments are sealed off entirely. Hearth exists because we wanted an agent that doesn't just remember — it forgets, and what it forgets (and what it keeps) shapes who it becomes.

What it does

Hearth is an agentic memory layer where memories have lifecycles:

  • Decay: every memory moves through six temporal bands (active → glimmer → beacon → half_sunk → deep → nebula) based on when it was last touched. A memory in deep cannot be recalled — until someone touches it and it rises back.
  • Anchors: tier 0–3 anchors slow or halt decay. A tier-3 memory never sinks.
  • Sealed records: private entries that exist in the database but never appear in recall results, cannot be searched, and have no embeddings computed. Zero-leakage is a hard evaluation constraint.
  • Supersession: when a memory replaces an older one, the old version is excluded from recall but preserved for history.
  • Keys + Semantic dual surfacing: an incoming message first triggers exact keyword matches, then a cosine-similarity search over bilingual embeddings. Both channels feed into a unified surfacing decision.
  • Agent choice: surfaced memories are hooks, not answers. The agent decides which ones to open — touch resets the decay clock; skip doesn't.

The browser demo shows the full loop: speak → surfaced memories with channel attribution → save a new memory (pending → ready) → watch the constellation shift.

How we built it

  • CockroachDB Cloud stores all memories, embeddings, sessions, and recall configuration. The schema uses scope-partitioned vector indexes for cosine recall, with lifecycle/sealed/decay filters applied post-ANN.
  • CockroachDB Cloud Managed MCP provides read-only operational health checks: schema verification, embedding coverage, decay band distribution, session overlay status, and vector index visibility — without exposing memory bodies or write access.
  • Jina Embeddings v3 (1024-dim, L2 normalized, separate passage/query tasks) powers bilingual retrieval. The threshold (0.3157) was calibrated on a 48-query train set and frozen; validation and test sets were never used for tuning.
  • AWS Fargate (ECS) runs the asynchronous embedding worker with lease-based job claiming and stale-write protection, plus the public demo app. Jina API keys live in AWS Secrets Manager, never in code or environment files.
  • HearthEval v1.1 is a frozen 128-query, 15-category evaluation suite covering retrieval, no-hit, exclusion, sealed leakage, supersession, decay-band probes, cross-language, and body-detail recall. Test results: Hit@3 0.903, No-hit 1.000, forbidden 0, sealed leakage 0, cross-language 7/7, P95 492ms.

Challenges we ran into

  1. Bedrock account-level gate: our AWS account requires an allowlisting review before any InvokeModel call. We pivoted to Jina within 24 hours and verified the full pipeline on Fargate.
  2. CockroachDB vector index filter constraints: WHERE clauses outside the index prefix disable the vector search path. We learned this through EXPLAIN and moved all non-prefix filters to post-retrieval, adding EXPLAIN evidence as a CI-level gate.
  3. Bilingual threshold calibration: English and Chinese embeddings occupy different similarity ranges. A single global threshold risks either missing Chinese recalls or flooding English results. The train-only calibration found a point that holds for both; the 7/7 cross-language test set confirms it, but we report the val set's one false positive (q051, karaoke → rice balls, score 0.34) rather than tuning it away.

Accomplishments that we're proud of

  • Sealed entries have zero leakage across all 128 evaluation queries. A sealed memory has no embedding, no recall eligibility, and no body exposure — not even through side-channel similarity to neighboring entries.
  • The evaluation suite is frozen and reproducible: same threshold, same data, same queries. We don't cherry-pick results.
  • Graceful degradation is real: when the semantic provider times out, the API returns keys-only results with semantic_status: timeout instead of failing. This happened in local development (Jina unreachable from our network) and the system kept working.
  • Cross-language recall works: a Chinese question retrieves an English memory (and vice versa) at 100% accuracy on the test set.

What we learned

That "memory" for an agent isn't a retrieval optimization — it's a design commitment. Deciding what fades, what stays sealed, what the agent can choose to skip — these are identity decisions, not engineering tradeoffs. The hardest part wasn't building the vector search; it was writing 75 fictional memories that felt real enough to evaluate honestly.

What's next for Hearth

  • Semantic recall integration into the private Hearth instance (the bilingual threshold provides a starting point)
  • Keys frequency-based noise suppression (high-skip keys in real conversations need dampening)
  • Voice interface for the physical embodiment (Stack-chan robot with cloud TTS — listen/speak loop working as of submission)

Built With

Share this project:

Updates

Submission history