Inspiration
Through my ongoing work with ChatGPT, and later with Codex, I began accumulating a huge amount of interaction history. These sessions contained ideas, decisions, mistakes, research, useful ways of working, unfinished directions, and things worth returning to later.
Over time, I started to see this data as a gold mine. It is especially valuable to me because it reflects my own tasks, interests, projects, questions, and way of thinking. Yet it was almost impossible to make practical use of all that accumulated experience. The sessions turned into an enormous stream of raw information.
For a long time, I had been thinking about a system that would let me return to these sessions, find important fragments, and gradually deepen my interaction with AI. Some ideas arrive too early. At the time of a conversation, the right models, tools, or level of agent capability may not exist yet. Later, the situation changes, but the idea is already buried somewhere in the message history.
I wanted to preserve the ability to return to it, see the original context again, understand the path of reasoning, and continue the work with a new level of AI capability.
This idea first appeared when I was still working mostly with ChatGPT. After I moved to Codex, it began to take a more concrete form. As I learned more about agentic work and started running increasingly long sessions, I realized that Codex was already capable enough for us to begin building such a system together.
That is how aoa-session-memory began.
What it does
At first, the task was fairly practical. I wanted to quickly restore context for the agent after a long session had been compacted, find the relevant events, and continue working without losing important decisions, errors, and verification results.
This problem becomes especially visible in large tasks. During a single session, an agent may explore a repository, try several approaches, abandon some of them, change the architecture, write code, run tests, and revise earlier conclusions several times. Usually, only part of that trajectory survives.
While developing the system, I noticed that agent sessions contain recurring entities and event types. There are goals, tasks, decisions, tools, skills, MCP servers, commands, errors, checks, results, and relationships between them. These elements can be defined, distinguished, and linked to the provenance of every meaningful conclusion.
From that point, the project quickly began growing beyond the original task of context recovery.
Today, aoa-session-memory preserves agent sessions together with their internal structure and the provenance of significant events. It builds several forms of navigation and analysis over this data.
The system preserves the original trajectory of the work, forms segments and episodes, identifies entities and events, and provides exact, semantic, temporal, and graph-based retrieval routes. These routes are currently at different levels of maturity.
The system aims to preserve a path from important results back to the specific session, event, command, tool response, or repository state they came from. Derived representations do not replace the original events; they serve as navigation back to them.
A lot of attention goes into preserving provenance accurately. A skill may be mentioned in a session, but the mention alone says nothing about whether it was actually used. The skill may have been available to the agent, selected, read, applied, completed, and verified. Each of these states has a different meaning. The same applies to tools, MCP, decisions, and command results.
This level of detail makes it possible to use past experience for meaningful evaluation of the system. Session history can show which skills were actually used, where they helped, where the agent deviated from the procedure, which errors keep recurring, which tools introduce unnecessary complexity, and which changes improve later work.
Continuing work after context compaction or moving into another session remain important use cases, but the project now covers much more.
Accumulated experience can be used for reflection over past and active sessions, building evals, and testing and improving skills, MCP, tools, prompts, agent workflows, automations, documentation, and code.
The system also connects the development process with the state of the repositories being built. It is possible to trace which session produced a decision, which alternatives were considered, why a particular path was chosen, how it was implemented in code, and how the result was verified.
Another direction is the creation of personal datasets. Raw transcripts contain too much repetition, error, temporary hypotheses, and unsupported claims. aoa-session-memory provides a foundation for gradually labeling, evaluating, and selecting useful experience. Over time, this data can be used to build verified collections of examples for specialized agents, personal models, and other forms of AI adaptation.
For me, this is one of the project’s main long-term goals. Every new session can become part of a cumulative process in which past experience helps evaluate and improve future work.
How we built it
The project was developed through long Codex sessions using GPT-5.5 and 5.6 Sol. Most of the work on the current architecture was done with GPT-5.6 Sol. Codex participated in architectural research, writing and reworking code, testing, analyzing failures, creating skills, writing documentation, and building verification mechanisms.
Challenges we ran into
Many architectural decisions emerged after specific failures were discovered in real sessions. For example, a semantically similar episode could appear to answer a question even though it did not contain the evidence required to support the answer. An old projection could remain technically valid after the classification logic had changed. History inherited by a child session could be mistaken for work performed after the fork.
Accomplishments that we're proud of
At the same time, aoa-session-memory was used during its own development. The same long-running agent sessions that motivated the project also became material for testing and further calibration.
What we learned
During this process, I learned how easily accumulated memory can distort the past when preserved events, interpretations, and the current state of a project are mixed together. A significant part of the work therefore went into data provenance, trust boundaries, projection freshness, and the ability to verify every important conclusion against the original events.
What's next for aoa-session-memory
I want to continue developing the path from accumulated agent experience to reflection, evals, improved tools, and personal datasets. In the long term, this should make it possible to grow increasingly specialized agents that gradually develop through verified experience from their own real work.
Log in or sign up for Devpost to join the conversation.