Mnemcore

GIF
liquid glass logo
login form
Overview for an org
Video library
video watch page
add a written or voice note
ask about this video
search interface for embeddings (video notes, moments)
hybrid search for notes with link to evidence video moments
usage page for an org

Inspiration

I have spent much of my life moving between two worlds: building software and playing rugby.

As a full-stack engineer, I have worked on complex systems for defense, optimization, and artificial intelligence. As a rugby player, I have experienced how teams actually learn: through film sessions, coaching notes, conversations, repeated mistakes, and small insights collected over an entire season.

The problem is that most of that knowledge disappears.

A coach may recognize the same defensive breakdown across five different matches, but the evidence remains scattered across hours of video and hundreds of isolated notes. An analyst may document an important pattern, but nobody remembers the exact game, moment, or timestamp when it happened. When a player leaves, a coach changes teams, or a season ends, much of the organization’s accumulated understanding leaves with them.

Video platforms help teams store footage. They do not help teams remember what they learned from it.

That realization inspired Mnemcore: a semantic memory system for video.

The name comes from the idea of a memory core—a shared intelligence layer that turns an organization’s videos, observations, and decisions into knowledge that can be searched, questioned, and continuously built upon.

We began with sports because the problem is visible there every day. But the larger vision applies anywhere important knowledge is trapped inside video: education, research, training, operations, interviews, and creative work.

Mnemcore is our attempt to give organizations something they have never truly had before: memory.

What it does

Mnemcore transforms video and human-authored observations into a searchable, conversational knowledge system.

Teams upload videos and capture timestamped notes while reviewing them. Mnemcore embeds those observations, connects them to their original moments, and makes the organization’s accumulated knowledge searchable across every video—not just within one recording.

A coach can ask:

“How have teams been disrupting our exit plays?”
“Where have we repeatedly lost possession near the sideline?”
“What patterns appeared across our last five matches?”
“Show me every example of pressure affecting our decision-making.”

Instead of generating a generic answer from an entire video, Mnemcore retrieves the most relevant evidence from the organization’s own knowledge base. Answers are grounded in the coaches’ observations and linked back to the exact videos and timestamps where the evidence came from.

This creates a continuous loop:

Watch → capture → retrieve → understand → improve.

The goal is not simply to summarize video. It is to preserve the meaning that people discovered while watching it.

Over time, Mnemcore can identify recurring concepts across videos and surface them as organizational signals. A single comment in one match may be an observation. The same idea appearing across several matches may be a pattern. That pattern becoming more frequent may be something the organization needs to act on.

Mnemcore turns disconnected notes into shared organizational knowledge.

How we built it

Mnemcore is built as a modern web platform using Vue, TypeScript, Tailwind, FastAPI, PostgreSQL, Supabase, pgvector, and cloud video infrastructure.

The intelligence layer combines several forms of retrieval:

Semantic vector search for conceptually related observations
Lexical search for names, terminology, and exact phrases
Timestamp and video constraints for precise questions
Reciprocal rank fusion to combine multiple retrieval strategies
OpenAI models to synthesize grounded answers from retrieved evidence

Every note is attached to an organization, video, and timestamp. We embed not only the note’s text but also useful temporal context, allowing questions about specific moments or phases of a video to retrieve more relevant evidence.

For long videos, we experimented with hierarchical representations such as fixed time windows and summaries. These structures help Mnemcore understand broader context without forcing a model to process an entire match or recording every time a user asks a question.

Our question-answering pipeline is intentionally evidence-first:

Understand the user’s question and explicit constraints.
Search the organization’s notes and knowledge representations.
Retrieve a small set of relevant, timestamped evidence.
Ask an OpenAI model to answer using only that evidence.
Return the answer with links to the original moments.

We also began developing an organizational signals pipeline. Notes are represented as embeddings, recurring semantic groups are identified, and an OpenAI model synthesizes each evidence cluster into a concise explanation. Clusters that appear across multiple videos can become organization-level signals rather than one-off observations.

Codex has been central to the development process. We used it not only to generate code, but to reason through architecture, build evaluation cases, inspect retrieval failures, simplify overengineered systems, and rapidly iterate across the frontend, backend, database, and AI pipeline.

Mnemcore itself has become an example of the type of collaboration we want to enable: human experience combined with AI that can preserve, organize, and extend that experience.

Challenges we ran into

The hardest part was not generating an answer. It was generating an answer that deserved to be trusted.

Early versions of Mnemcore could produce convincing responses that were not always fully supported by the underlying notes. A model might correctly understand the general context but attach the wrong player, infer an unsupported comparison, or cite evidence that only partially justified the claim.

In sports analysis, a confident but incorrect answer is worse than no answer.

We had to rethink the system around evidence rather than fluency. We introduced stricter retrieval boundaries, timestamped citations, evaluation datasets, and rules requiring answers to remain grounded in the retrieved notes.

Another major challenge was question routing.

Users can ask for a timestamp, an event, an entity, a comparison, a sequence of plays, a full summary, or something that cannot be answered from text notes at all. We initially attempted to classify every question into a detailed taxonomy and generate a specialized retrieval plan.

The system became too complicated.

Our evaluations showed that the plans were structurally valid, but the classifications often failed in subtle ways. A question about who caught a touchdown could be interpreted as either an entity question or an event question. A request for a final score could be treated as a summary even though it referred to a specific event outcome.

We learned that elegant taxonomies do not automatically produce reliable products.

We stepped back and simplified the architecture. Explicit constraints such as timestamps are handled deterministically. Most other questions begin with strong hybrid retrieval over the user’s actual notes. The model’s primary responsibility is to explain the selected evidence—not invent an elaborate search strategy before seeing it.

We also faced the challenge of scale and stability.

An organization may eventually contain thousands or millions of observations. Clustering those embeddings can reveal powerful patterns, but clusters can shift as new data arrives. A pattern must also span multiple videos before it truly represents organizational knowledge.

We are designing the system so signals are reproducible, traceable to their source evidence, and honest about their strength.

Perhaps the most important challenge was resisting the temptation to build everything at once. At different points, we explored entities, events, topics, chapters, segments, summaries, planners, routers, graphs, and multiple layers of generated metadata.

Many of those ideas remain valuable, but each additional abstraction introduces cost and another place where meaning can be lost.

The journey of building Mnemcore has largely been a journey toward simplicity.

Accomplishments that we're proud of

We built a working product that can transform hundreds of timestamped human observations into an organization-wide semantic knowledge base.

Our development corpus contains more than 800 timestamped notes across multiple full-length sports videos, representing tens of thousands of words of real analysis. Users can search those observations across videos, ask natural-language questions, and return to the exact moments supporting an answer.

We are especially proud that Mnemcore does not treat human notes as secondary metadata. The coach’s observation is the core unit of knowledge.

AI does not replace the analyst. It makes the analyst’s understanding reusable.

We built a complete pipeline across video ingestion, note capture, embeddings, hybrid retrieval, organization scoping, evidence selection, answer generation, and timestamp navigation.

We also created repeatable evaluations for our question-answering architecture. Those evaluations exposed weaknesses that would have been easy to hide behind impressive demos. Rather than optimizing only for answers that sounded good, we began measuring whether the correct evidence was retrieved and whether each claim was actually supported.

One of our most important accomplishments was recognizing when our own system had become overengineered—and having the discipline to simplify it.

We moved from attempting to pre-understand everything inside every video to a more practical principle:

Store what people observed, retrieve what matters, and reason only from the evidence.

That principle has made Mnemcore faster, more understandable, and more trustworthy.

What we learned

We learned that memory is not the same as storage.

Organizations already have video libraries, folders, documents, chat histories, and databases. The problem is not that information is missing. The problem is that the relationships between those pieces of information are invisible.

Knowledge becomes valuable when people can recover it at the moment they need it.

We learned that embeddings are powerful, but retrieval is a product discipline rather than a single algorithm. Semantic similarity, keyword matching, timestamps, organization boundaries, evidence diversity, and source quality all matter.

We learned that an AI system should know the boundary between interpretation and invention. A good answer is not simply plausible. It must be attributable to evidence.

We learned that humans and AI contribute different kinds of intelligence.

A coach recognizes the importance of a moment. A player feels what changed on the field. An analyst notices a repeated structure. Mnemcore preserves those judgments, connects them across time, and makes them available to the rest of the organization.

We also learned that organizational knowledge often emerges gradually.

One note is a thought.

Several related notes are a theme.

The same theme appearing across different videos is a signal.

A signal changing over time can become a decision.

This progression—from observation to organizational intelligence—is becoming the foundation of Mnemcore.

Finally, we learned that the best AI architecture is not necessarily the one with the most agents, prompts, or generated metadata. It is the architecture that consistently helps a real person reach the right evidence and make a better decision.

What's next for Mnemcore

The next stage of Mnemcore is to evolve from a system that answers questions into a system that helps organizations notice what they are not yet asking about.

We are building organization-level signals that identify recurring patterns across videos, track whether those patterns are increasing or decreasing, and link every insight back to the underlying evidence.

We also plan to introduce Focuses: goals defined by a coach, team, or organization that Mnemcore continuously evaluates against new observations.

A coach might define a focus such as:

Improve our decision-making under pressure.

As new matches and notes are added, Mnemcore could identify relevant evidence, show whether the organization is improving, and surface the moments that most clearly demonstrate progress or regression.

This moves the product toward a continuous learning loop:

Observe → orient → decide → act → remember.

Beyond sports, we see Mnemcore becoming a semantic memory layer for any organization that learns through video.

A research team could connect findings across recorded experiments.

A company could preserve knowledge from training sessions and interviews.

An educator could identify recurring misconceptions across lectures and student discussions.

A creative team could search years of footage by meaning rather than filenames.

Our long-term goal is ambitious but simple:

Every organization should be able to remember what it has seen, understand what it has learned, and build upon that knowledge instead of repeatedly starting over.

Mnemcore is building the memory layer that makes that possible.