What Inspired This Project
Large language models are powerful, but they are fundamentally stateless. Every conversation starts from zero. Every document you shared last week is gone unless you paste it again. This is not a minor inconvenience. It is an architectural limitation that prevents AI from functioning as a reliable long-term knowledge tool.
Traditional Retrieval Augmented Generation systems attempt to solve this by storing raw text chunks in a vector database and retrieving them by semantic similarity at query time. But these systems treat all stored data as permanently and equally relevant. They do not model the fact that knowledge fades when it is unused. They do not distinguish between a concept you discussed yesterday and a concept you discussed six months ago. They do not connect related ideas across different documents. And they rely entirely on embedding similarity, which misses exact terminology and relational context.
We built ContextOS because we wanted AI agents to have real memory. Not a searchable document store. Not a flat vector database. A structured, living knowledge system where memories are extracted once, connected through relationships, ranked by multiple signals, and allowed to decay naturally over time the way human memory does.
The question that drove the project was simple: what would it take to give an AI agent a memory system that actually works the way memory should work?
The Problem We Solve
There are three specific problems with how AI memory works today.
First, statelessness. Language models do not retain information between sessions. If you explain your codebase to an AI on Monday, it does not know anything about your codebase on Tuesday. Users work around this by repeatedly re-pasting context, which wastes tokens and time.
Second, context pollution. Existing RAG systems store documents as raw text chunks with no structure, no importance weighting, and no expiration. Over time, the retrieval index fills with outdated, irrelevant, or redundant information that competes with current knowledge for space in the context window.
Third, single-signal retrieval. Most systems retrieve context using only vector similarity. This works well when the user's wording closely matches the stored text, but fails when the user asks about a concept using different terminology, or when the relevant answer spans multiple related concepts that are not semantically adjacent in embedding space.
ContextOS addresses all three problems through a unified architecture that combines structured concept extraction, a decay-aware memory model, and triple-path retrieval.
How We Built It
Architecture Overview
ContextOS separates the system into four layers: Ingestion, Knowledge Storage, Retrieval, and Interaction. Documents and conversations flow through the ingestion layer exactly once. The extracted knowledge serves every future query.
Documents Conversations
| |
v v
Document Ingestion Chat Retrieval (real-time)
| |
| Parse | Query Analysis
| Normalize | 3 Parallel Searches
| Detect Domain | - Vector (semantic)
| AI Extraction | - Text (lexical)
| (single call) | - Graph (relational)
| Generate Embeddings | Merge and Score
| Store Buckets | Assemble Context
| Build Relationships | Inject into Prompt
| |
v v
+--------- Knowledge Store ---------+
| Buckets (concepts with decay) |
| Embeddings (1536-dim vectors) |
| Relationships (typed edges) |
| Raw Text (original source) |
| Conversations (message archive) |
+-----------------------------------+
| |
v |
Background Scan |
(every 4 hours) |
| |
| Decay strength |
| Trigger reminders |
| Cold storage migration |
| |
v v
Chat Buffer -----> Async Conversation
Ingestion
|
v
Extract Concepts
Generate Embeddings
Merge into Knowledge Store
The Ingestion Pipeline
When a document is uploaded, the system runs a seven-step pipeline. The critical engineering decision is that the AI extraction step happens exactly once for the entire document, not once per chunk.
Step 1: Parse and Normalize. The system detects the file type (PDF, DOCX, markdown, text, or code) and extracts clean text. Formatting artifacts such as HTML tags, markdown syntax, and excessive whitespace are removed. The document is tagged with metadata including source type, filename, timestamp, and detected domain. A content hash prevents duplicate processing. This entire step is local with zero API calls.
Step 2: Structural Analysis. For large documents that may exceed the AI model context window, the system identifies structural boundaries such as headings, sections, and paragraphs. Domain-specific chunk strategies handle different document types:
| Document Type | Splitting Strategy |
|---|---|
| Research papers | Abstract, introduction, methods, results, discussion |
| Books | Chapters |
| Textbooks | Chapters and subsections |
| Documentation | Sections by heading |
| General text | Paragraph boundaries |
| Notes | Timestamp or topic boundaries |
This step is also entirely local.
Step 3: AI Concept Extraction. The full document text is sent to the language model in a single call. The model is constrained to output a JSON array of conceptual nodes. Each node contains a label (2 to 5 words), a contextual definition (2 to 3 sentences explaining why the concept matters in this document), a type (problem, decision, fact, entity, event, preference, or code), an importance score from 1 to 10, and a list of related concepts.
For documents that fit within the model context window (under 200K tokens), this is a single Bedrock call. For the rare documents that exceed the window, the text is split into sections from Step 2, each section is processed separately, the model sees previous section outputs when processing subsequent sections, and a final deduplication pass merges all results.
{
"label": "Group Relative Policy Optimization",
"definition": "Core training method of LongStraw that enables
million-token RL post-training on fixed GPU budgets by comparing
responses within prompt groups, eliminating the need for a
separate critic model.",
"type": "fact",
"importance": 10,
"related": ["LongStraw", "RL post-training", "GPU budget"]
}
Step 4: Validation. Every concept returned by the model passes through a validation layer that rejects empty labels, labels that form full sentences, labels with trailing connector words, definitions that only repeat the label, definitions that are too short, and invalid concept types. Importance is clamped to the 1 to 10 range. Duplicate normalized labels are removed. Related concepts that do not exist in the accepted set are dropped.
Step 5: Raw Text Storage. The complete original text is stored in CockroachDB. Documents exceeding database storage limits are stored in S3 with a reference key. The raw text is linked to the document ID for bidirectional lookup: from any concept you can find its source, and from any source you can find all concepts it produced.
Step 6: Embedding Generation. Each validated concept is converted into a 1536-dimensional dense vector by Amazon Titan Embeddings. The embedding input combines the label, definition, and significance into a single text. All concepts are sent in parallel using Promise.allSettled. Ten concepts produce ten concurrent API calls that complete in roughly one second total, not ten sequential calls.
Step 7: Bucket Creation and Relationship Storage. For each concept, the system checks whether a matching bucket exists using normalized key comparison. The normalization process converts the label to lowercase, removes punctuation, strips generic modifier words like "system" or "approach", splits into words, sorts them alphabetically, and joins them into a stable key. If a matching bucket already exists, the new concept is added as a bucket item and the strength is increased. If no match exists, a new bucket is created. Relationships are stored between connected concepts based on the related field from the AI output.
The Memory Model
Every memory bucket maintains a strength value between 0.0 and 1.0. Strength decays over time based on importance and is reinforced when the memory is accessed.
The decay formula is:
$$ strength(t) = initial\_strength \times (1 - decay\_rate)^{days\_since\_last\_access} $$
Decay rates are assigned based on importance.
| Importance | Range | Decay Rate | Behavior |
|---|---|---|---|
| High | 8 to 10 | 0.10/day | Persists for weeks |
| Medium | 5 to 7 | 0.15/day | Default curve |
| Low | 1 to 4 | 0.20/day | Fades in days |
Strength categories determine system behavior:
| Category | Range | Behavior |
|---|---|---|
| Strong | above 70% | Always included as retrieval candidate |
| Fading | 40 to 70% | Included only if relevant to query |
| Critical | below 40% | Triggers proactive reminder to user |
| Forgotten | below 10% | Excluded from search, moved to cold storage |
When a memory is accessed and used in context injection, its strength is reinforced:
$$ new\_strength = old\_strength \times 0.7 + 1.0 \times 0.3 $$
This produces a feedback loop. Memories the user actually interacts with stay strong. Memories that are never referenced fade naturally and eventually move to cold storage backed by S3. If a future query matches a cold memory through vector similarity, it is reactivated by promoting it back to warm storage with strength reset to 50%.
A background scan runs every 4 hours to compute current strength for all memories, trigger proactive reminders for critical memories, and migrate forgotten memories to cold storage.
The Retrieval Pipeline
When a user sends a chat message, the system runs three parallel searches and merges the results.
User Query
|
+-------------+-------------+
| | |
v v v
Vector Text Search Graph
Search (lexical) Traversal
(semantic) (relational)
| | |
+-------------+-------------+
|
v
Result Merging
|
v
Relevance Scoring
|
v
Forgetting Budget
|
v
Context Assembly
|
v
AI Model Response
Vector search embeds the query into 1536 dimensions via Titan and performs approximate nearest neighbor search against stored concept embeddings using cosine distance with CockroachDB's distributed vector index. Returns the top 20 most semantically similar buckets. This provides breadth: it finds conceptually related memories even when the user uses different wording.
Text search extracts key noun phrases from the query and runs SQL pattern matching against bucket labels and definitions. Up to 5 queries run in parallel, one per key term. This provides precision: it catches exact terminology that embedding models can smooth over.
Graph traversal starts from the buckets found by vector and text search and follows relationship edges to connected concepts. Traversal is limited to 2 hops. This provides context: it surfaces related concepts that neither vector nor text search would find independently.
Results are merged and deduplicated by bucket ID. Each candidate tracks which search methods found it. Buckets found by multiple methods receive a relevance bonus.
Each candidate is scored using a weighted formula:
$$ relevance = 0.4 \times semantic\_score + 0.3 \times strength\_score + 0.3 \times recency\_score $$
Where the semantic score itself combines vector similarity (50%), text match (30%), and graph connections (20%). Recency score uses exponential decay: \( e^{-0.1 \times hours\_since\_last\_access} \).
Scores are normalized with softmax and sorted descending. Tie-breaking follows this order: higher strength first, then more recent access, then higher importance.
The top N memories are selected up to the forgetting budget (default 20), with a diversity constraint that prevents more than 60% from a single source. The selected memories are formatted into a structured context block and injected into the system prompt.
Conversation Memory
During real-time chat, no extraction or storage happens. The system only retrieves existing memories and injects them into context. The user gets a response in approximately 200 milliseconds. The conversation is buffered.
After the conversation ends, triggered by explicit session end, 15 minutes of inactivity, or exceeding 50 messages, the buffered messages are assembled into a conversation log and sent to the language model to extract 3 to 10 important concepts. These follow the same storage pipeline: embedding, bucket creation or merging, and relationship building.
This two-path design keeps chat responses fast while still capturing important conversational knowledge as persistent memories.
Key Technical Decisions
Single extraction call instead of per-chunk extraction. Traditional RAG systems send each text chunk separately to the AI model. This is expensive and severs contextual relationships between chunks. ContextOS sends the full document (or structural sections for very large documents) in one call, preserving cross-section context and reducing API costs to one Bedrock call per document for most uploads.
Triple-path retrieval instead of vector-only search. Vector search alone misses exact terminology. Text search alone misses semantically related concepts. Graph traversal alone misses unconnected concepts. Running all three in parallel and merging results produces better recall and precision than any single method. The parallel execution keeps latency under 500 milliseconds.
Memory decay instead of permanent storage. Existing systems treat all stored data as equally relevant forever. This leads to context window pollution where outdated information competes with current knowledge. The exponential decay model based on importance ensures that high-value knowledge persists while low-value knowledge fades naturally.
Single CockroachDB for relational, vector, and graph data. Instead of running separate databases for different data types, ContextOS uses CockroachDB for everything: relational tables for metadata, distributed vector indexes for embedding search, and a relationships table for graph edges. This eliminates operational complexity and enables atomic transactions across all data types.
Asynchronous conversation ingestion. Extracting concepts during live chat would add latency to every response. By deferring extraction to after the session ends, chat responses stay fast while conversational knowledge still becomes persistent.
Challenges We Faced
Large document handling. Documents that exceed the AI model context window required a section-based extraction strategy where the model sees previous section outputs when processing subsequent sections. Getting the deduplication pass right across section boundaries was the most complex part of the ingestion pipeline.
Normalized key deduplication. Deciding when two concepts are the "same" concept is not straightforward. A concept labeled "Group Relative Policy Optimization" and one labeled "GRPO training method" should merge into the same bucket. The normalization pipeline (lowercase, remove punctuation, strip modifiers, sort words, join) handles most cases, but edge cases around abbreviations and paraphrasing required careful testing.
Retrieval relevance tuning. Balancing the weights between semantic score, strength score, and recency score required testing with real queries against real document sets. Abstract questions (like "What is the main contribution?") behave differently from specific factual queries (like "What is the decay rate for importance 8 concepts?"), and the scorer needed to handle both patterns well.
Cold memory reactivation. Deciding when and how to bring forgotten memories back into the active set required balancing between keeping the search space clean and not permanently losing knowledge. The reactivation threshold of 50% strength on match provides a reasonable middle ground.
What We Learned
Building a persistent memory system requires thinking about knowledge as a living structure, not a static index. The decay and reinforcement model fundamentally changes how the system behaves over time. Memories that matter to the user stay strong. Memories that do not matter fade. The system becomes more useful the longer it is used, because the knowledge graph reflects the user's actual needs rather than accumulating everything indiscriminately.
The single-extraction-call architecture proved that you do not need per-chunk AI processing to build effective retrieval. Sending the full document and letting the model identify the important concepts produces better results than blindly chunking text and embedding each chunk separately.
Triple-path retrieval was more valuable than expected. In testing, graph traversal consistently surfaced relevant concepts that neither vector nor text search found on their own. The combination of all three methods produced meaningfully better retrieval than any single method alone.
Technology Stack
| Layer | Technology |
|---|---|
| Backend | TypeScript, Node.js, Express |
| Frontend | React, Vite, TailwindCSS |
| Database | CockroachDB (relational, vector, and graph storage) |
| AI Models | Amazon Bedrock (Nova Pro for extraction, Titan for embeddings) |
| Infrastructure | AWS Lambda, S3, SQS, API Gateway, CloudWatch |
| Visualization | Custom WebGL and Canvas knowledge graph renderer |
Log in or sign up for Devpost to join the conversation.