ctxindex

Inspiration

This started because I was trying to make my Hermes agent actually useful with my email and other accounts.

I had connected a few existing command-line tools, including gog. It worked for simple things, but I literally couldn't get multiple accounts working reliably. Search was split between different tools, they didn't all support the same features, and Hermes would sometimes just miss things.

I started ctxindex as a better solution for that specific problem. Then it became obvious that hard-coding every integration into one huge tool would eventually create the same mess again.

I like how Pi approaches extensions: keep the core small and let people add what they need. I wanted to take a similar approach here. ctxindex should provide the common foundation, while Extensions handle new providers and new kinds of context.

What it does

ctxindex, pronounced “context index,” gives agents one command-line tool for accessing email, calendars, files, and other context.

It currently supports Gmail, Outlook, Google Calendar, Microsoft Calendar, local directories, and additional sources provided by Extensions.

You can connect multiple accounts and put their Sources into Realms such as personal or work. An agent can search across them, get a result, and then use its ctx:// reference to retrieve the complete message, event, file, or thread.

The agent doesn't need a special integration. If it can run a shell command, it can use ctxindex. I built it for Hermes, but it works in the same way from Codex, Claude Code, or another agent.

Everything stays local. Google, Microsoft, and your files are still the real source of the data. ctxindex only keeps a local index and cache.

It can also create or update email Drafts, but it never sends mail. I wanted that boundary to be very clear.

How I built it

ctxindex is mostly TypeScript and runs on Bun. It has a local daemon, SQLite storage, a CLI, Google and Microsoft support, an Extension SDK, tests, and the documentation website.

Extensions can add new types of context, providers, authentication, actions, documentation, and assets. They can be installed from npm, Git, a local package, or a Catalog. The official Catalog works as the marketplace.

I built most of the project with GPT-5.6. I started the work in Pi and later continued in Codex.

In Codex, I used one huge session as a meta-session. I stayed with the root agent to plan what we were doing and make decisions. Smaller subagents handled focused implementation, testing, research, and review tasks. It was a bit like running a small development team from one conversation.

Codex was also useful outside the editor. Its browser and computer tools helped me work through Google Cloud Console and Microsoft Entra while setting up real OAuth applications. Those consoles are massive, so that saved a lot of time.

An early version of ctxindex existed before Build Week. The work from the submission period is visible in the commit history and dated agent sessions. I also don't want to pretend that the Pi work was done in Codex. It wasn't. Both used GPT-5.6, but they were separate parts of the process.

Challenges I ran into

Multiple accounts were the original problem, and they stayed one of the hardest parts.

The Gmail and Microsoft APIs themselves weren't really the main difficulty. The difficult part was making accounts, providers, permissions, search results, files, events, and email all fit together without adding a special case for everything.

OAuth was also painful. Google and Microsoft use different terminology, different consoles, and different rules. Something that works for a personal account may fail completely for an organization account.

The Extension system took several attempts too. I wanted Extensions to be powerful, but I didn't want them to turn ctxindex into an unsafe “run arbitrary commands” system.

I also spent a surprising amount of time on boring things that agents really care about: consistent output, stable references, exit codes, errors, cancellation, packaging, and making sure the installed CLI behaves like the development version.

The public demo was another challenge because most things I use ctxindex for contain private email, calendars, or files. I couldn't just record my normal workflows and upload them publicly. The demo is intentionally limited for that reason, and I also made a public no-login example using GitHub issues.

Accomplishments that I'm proud of

I'm mostly proud that it became a real tool and not just something that only works in the demo video.

The CLI is published on npm. The repository is public and MIT licensed. There are public docs, examples, an Extension SDK, an official Catalog, and a demo that doesn't require someone to connect their email.

I'm also proud of the Extension support. That was the point where ctxindex stopped being only a solution for my Hermes setup and became something other people could build on.

And honestly, I'm proud of the codebase. It became much larger than I expected, but it still has clear boundaries, tests, written decisions, and checks for things I didn't want to slowly break.

What I learned

I learned that giving an agent access to your context isn't just a search problem.

The agent needs to know which account something came from, where the original data lives, whether a result is complete, how to retrieve it again, and what it is allowed to change. Without that, even a very capable model ends up guessing.

I also learned that using many agents only works when their jobs are narrow. If several agents all change whatever they want, the result becomes a mess. Keeping planning and review in the main session while giving subagents specific pieces of work was much better.

What's next for ctxindex

The immediate plan is to keep using it from Hermes and fix whatever annoys me in real use.

I want to add more providers and context types, improve search across Sources, grow the official Catalog, and make Extensions easier to build and publish.

OAuth setup also needs to become less painful. Right now it works, but it is easily the worst part of getting started.

I don't know exactly how large ctxindex will become, but I want to keep the main idea simple: one reliable way for agents to access the context you already have.

Built With

• codex
• gpt5.6