Cogitator

Inspiration

Manually collecting information and feeding it into an LLM is tedious, and current assistants still can’t proactively help without constant prompting. Cogitator is our way to remove that annoying overhead so you can focus on absorbing what’s important instead of doing repetitive setup work.

What it does

Cogitator is a macOS application that runs seamlessly on your device as a close partner for your daily life and work. It can “see” what you see, understand what you’re doing, and say what you need. In this new AI workflow, you don’t always have to ask first — it can anticipate and assist proactively.

How we built it

We built Cogitator on the macOS platform, using OCR, and embedding-based history system, and XAI APIs to deliver low-latency assistance for everyday typing. End-to-end latency is usually under 3 seconds. The system keeps track of your activity all day and can fetch related information from across your entire day’s context.

Challenges we ran into

High-performance LLM inference alone is not enough for a great user experience when users generate lots of screen recordings and images. Directly sending raw images often led to processing times over 10 seconds. Introducing OCR as a front-end step allowed us to turn visuals into text, dramatically reducing both latency and compute cost.

Accomplishments that we're proud of

We achieved low-latency OCR at around the 0.5 second level. Our embedding-powered history database supports retrieving related information from a full day of activity, enabling fast lookups of what you saw or worked on earlier, and making Cogitator feel like it truly “remembers” your day.

What we learned

Long-term memory, important memory, and highly related context storage and retrieval remain a valuable problem space. Through this project, we explored methods to maintain multiple flows of work while keeping compute costs low, performance high, and the user experience smooth and responsive.

What's next for Cogitator

Optimize, optimize, optimize. We want better understanding and better outputs: richer context modeling, smarter retrieval, and more proactive behavior. Our next steps focus on improving accuracy, robustness, and making Cogitator an even more natural part of everyday workflows.

Built With

• ai
• embedding
• http
• ocr
• swift