iSpy : MyMemories

• 

  Logo

Inspiration

Agentic Memory. From evals to generation improvement, local models, and leaf agents. Context is Key.

I built this project from an inspiration of what I saw in a Liquid AI x Cohere webinar where Machine Learning Leads are demonstrating what's already possible and already in development: "upgrades of things that weren't even possible 6 months ago". For example take a look at Arize/phoenix - AI observability for agent tracing and eval with 'PXI' , an engineering agent contextualized with needs within the evaluation. From where we stand, we have to keep building and creating. The human potential has no finish line and that applies to our bettering of AI.

What it does

'I-SPY' takes a user prompt and starts contextualizing the product about the user since day one. Most systems forget what you tell it, "iSpy : MyMemories" bases everything around YOU. It stores a split of semantic and episodic contextualization across stores of memory types, remembering that divide between preferences and 'events', 'tasks', 'signals' or cues, and considered 'offers'. The response is built around this retrieved contextualization. Appending what it learned with to a brief describing your TO-DOs, scheduled meetings, proposals you haven't yet decided on, and workflow or tool outcomes. A reflection agent pushes this further working on a AWS Lambda + Event Bridge Runtime Scheduling to consolidate, superseded memories, and summarize them appending tags and updating context as a background job.

Onto Actions: The app has come with some already integrated toolchains, with Automated Workflows attachable architecture where (n8n or langchain assumed) would run on its own clock, poll its integrations and append them to signals. By the time a user asks anything, the data's already sitting in signals.

How we built it

Everything built around the headline: remember→ enrich→ act → orchestrate.

I-SPY runs on CockroachDB as the single memory layer as structured entity tables (memories, calendar events, tasks, signals, offers) instead of one scattered vector store, so agents can query exactly what they need. Anthropic's Claude reads that memory, decides what matters, and chains tool calls to act on it. AWS Bedrock (Titan) generates the embeddings that make memory searchable in the first place. A reflection agent on AWS Lambda + EventBridge runs on its own clock, consolidating and cleaning up memory in the background. And an Express/Node backend ties it all together with one HTTP surface, one place the whole loop runs through.

Challenges we ran into

What should contextualize agentic memory? - Is it retrieval quality, is it the quality of ingestion at write, the reasoning power of the model, maybe data processing through pipelines before a response?

When I first started out, I used a fuzzy search with PostgreSQL in CRDB hydrated into a ground truth approach - using DynamoDB - for quick hits and a source of truth. While this approach had its merits, I saw the power of CRDB's strong transactional consistency. While this meant so much more architecture I started immediately, shifting out memories out of a scattered, independent single table vector search into structured tables, the agents and tools can query exactly. For the UI, the functionality, and logic - this meant reading information and out many more places, recording the data at different times i.e. the ledger, and evolving the SQL schema as well as function transactions as we went. This was my single-biggest challenge as one I continue to keep working at.

Accomplishments that we're proud of

Catching that distance is inversely useful for contradictions before it shipped since "mornings" and "afternoons" sit far apart in embedding space. It is why adding hybrid search for retrieval is a whole another solution and something I predicted as I built my first retrieval parameters. Additionally on our vector stores implementing the SQL Architecture and happing to build more queries for data extraction as necessitated was a point of focus I'm proud of. Lastly, the decision for wiring a reflection agent that runs on its own clock and actually keeps memory honest was what took an evolving memory entity a step further.

What we learned

• CockroachDB's strict type-checking inside SQL functions is not Postgres's.
• Retrieval Techniques per my use case, getting the composite ranking function right was how the app works.
• MCP is for interactive, agent-decided tool calls — not for scheduled batch work.
• The structuring of information that turns the idea from "how do I store vectors" to "how do I keep one consistent state layer an agent can trust on every turn." that the project is based around.

What's next for i-SPY : mymemories

• sqlc/PgTyped for SQL Cached Queries
• CRDB's CHANGEFEEDS for observable agentic infrastructure - gives a live CDC stream of every write
• Security/Encryption
• Use 07 hybrid_retrieval and evaluate
• Expand on Harness with PhoenixIntelligence

Built With

• agents
• amazon-web-services
• claude
• embeddings
• hybrid-fts
• javascript
• lambda
• python
• sql