Inspiration
As AI assistants become part of our daily workflow, I found myself repeatedly explaining the same context, reopening the same documents, and searching through browser history just to continue where I had left off. Current assistants are excellent at answering questions, but they forget everything once the conversation ends. They don't know what I've been working on, what is currently on my screen, or how I personally prefer to work.
At the same time, many existing solutions solve this by continuously uploading user data to the cloud, creating obvious privacy concerns.
I wanted to build something different.
My goal was to create a true personal memory agent that works more like human memory—remembering important experiences, forgetting unimportant ones, understanding my current context, and keeping those memories entirely on-device. Qwen Cloud would then be used only where intelligence is needed: reasoning over memories, planning actions, and answering questions.
That became iAletheia, named after the Greek word Aletheia, meaning truth or un-forgetting.
What it does
iAletheia is a privacy-first, on-device memory agent for macOS.
With the user's permission, it observes meaningful activity on the frontmost window using ScreenCaptureKit, Accessibility APIs and Vision OCR. Instead of storing screenshots, it extracts useful information, removes sensitive content and builds a structured local memory that can be searched later.
The assistant understands both my long-term memory and whatever is currently on my screen.
I can ask questions like:
- What was I researching yesterday about KV cache?
- Draft a reply to this email.
- Summarise what I'm reading.
- Have I solved this problem before?
One of my favourite features is Show Me.
Instead of simply explaining where something is, iAletheia analyses the current application, locates the correct UI element using Accessibility and OCR, and guides the user step-by-step directly on the screen using an overlay pointer. Rather than taking control of the computer, it teaches users how to complete tasks themselves.
Everything is stored locally, while Qwen Cloud provides reasoning, retrieval, web search, planning and natural language understanding.
How I built it
I built iAletheia entirely as a native macOS application using Swift and SwiftUI.
The application uses:
- ScreenCaptureKit to observe the active window
- Accessibility APIs to understand application UI
- Vision OCR for extracting visible text
- SQLite + FTS5 for local memory storage
- Apple Natural Language for on-device embeddings
- Qwen Cloud for reasoning, structured memory extraction, web search and response generation
Every observation first passes through a local privacy pipeline where sensitive information is filtered before any optional cloud reasoning occurs.
Queries are routed through different execution paths depending on whether they require live screen understanding, long-term memory retrieval, web search or a combination of all three.
Challenges I ran into
The biggest challenge wasn't building an AI assistant; it was deciding what should actually become a memory.
Capturing everything quickly becomes noisy and unhelpful, while storing too little makes the assistant forget useful context. Designing a selective memory pipeline that balances usefulness, privacy and performance required a lot of experimentation.
Another challenge was making the assistant understand the correct active window on macOS. Modern desktop environments can contain multiple displays, floating windows and overlays, so accurately identifying the user's intended context required combining ScreenCaptureKit, Accessibility APIs and OCR.
Building Show Me was another interesting challenge. I needed to reliably locate UI elements across different applications while keeping the guidance intuitive without automating user actions.
Accomplishments that I'm proud of
- Building a fully native macOS memory agent instead of wrapping a chatbot in a desktop application.
- Designing a completely local-first memory architecture where screenshots are never permanently stored.
- Combining long-term memory with live screen awareness.
- Building Show Me, allowing the assistant to teach users directly on the interface they're using.
- Integrating Qwen Cloud for intelligent reasoning while keeping personal memories on-device.
- Completing a working end-to-end prototype during the hackathon.
What I learned
This project reinforced that building useful AI assistants is less about larger language models and more about building better context.
I learned how difficult long-term memory actually is compared to traditional RAG systems. Memory requires deciding what deserves to be remembered, how memories evolve over time and when they should be forgotten.
On the engineering side, I gained a much deeper understanding of ScreenCaptureKit, Accessibility APIs, Vision OCR and building local-first macOS applications.
I also explored how Qwen Cloud can be used not simply as an LLM endpoint, but as the reasoning layer of a much larger memory architecture.
What's next for iAletheia
This prototype is only the beginning.
Next, I want to build:
- Memory consolidation that automatically merges related observations into higher-level knowledge.
- Smarter forgetting and memory decay inspired by human cognition.
- Support for documents, PDFs, IDEs and productivity tools.
- A richer memory graph connecting projects, people and ideas.
- Cross-device encrypted memory synchronisation while preserving local-first privacy.
- More advanced Show Me capabilities that can understand increasingly complex application workflows.
Log in or sign up for Devpost to join the conversation.