MemoryLens — OpenAI Memory Observability

Inspiration

AI memory makes assistants more useful, but it also creates a trust problem: when an assistant gives the wrong answer, teams often cannot see which memory caused it or why it was retrieved. We built MemoryLens to make AI memory transparent, testable, and safe.

What it does

MemoryLens is an OpenAI-powered observability dashboard for semantic memory. It shows the full retrieval trace: query, ranked memories, similarity scores, latency, and context sent to the AI.

It also provides Memory QA, regression detection, duplicate-memory detection, and PII Guard. Users can identify sensitive data such as emails, phone numbers, cards, API keys, and SSNs, then redact it and regenerate the memory embedding.

How we built it

We built the backend with FastAPI, SQLite, and JWT authentication. MemoryLens uses OpenAI’s text-embedding-3-small model to embed memories and search queries, then ranks memories using cosine similarity.

The frontend is a responsive vanilla JavaScript application styled with Tailwind CSS. Chart.js powers retrieval-quality and regression visualizations.

Challenges we ran into

The main challenge was making semantic retrieval understandable. Similarity scores alone do not explain an AI decision, so we designed the Retrieval Inspector to reveal the complete path from a user query to the retrieved context.

Privacy was another challenge. Memory persists by design, so we needed a way to identify and safely handle sensitive information without exposing it further in the interface.

Accomplishments that we're proud of

We are proud that MemoryLens is more than a retrieval demo. It gives teams practical tools to test memory quality, catch regressions, detect conflicting memories, and reduce privacy risk.

We are especially proud of the PII Guard workflow: detect, mask, redact, and re-embed—without requiring users to manually rebuild their memory index.

What we learned

We learned that AI memory needs the same level of evaluation and observability as AI models. Good memory is not only about retrieving relevant information; it is also about understanding why it was retrieved, measuring whether it continues to work, and keeping it safe.

What's next for MemoryLens — OpenAI Memory Observability

Next, we want to add team workspaces, scheduled memory-health reports, richer evaluation datasets, and OpenAI Responses API integration so users can trace retrieval all the way through to the final model response. We also plan to add configurable privacy policies and deployment-ready security controls.

Built With

• api
• chart.js
• css
• embeddings
• fastapi
• javascript
• jwt
• openai
• python
• sqlite
• tailwind
• text-embedding-3-small