Dockt

Dockt Logo

Inspiration

A workday’s important details are scattered across chats, documents, and browser tabs. You see a date, a request, or a promise on screen, switch tasks, and later have to dig around to find it again.

We wanted Dockt to make those fleeting moments searchable with answers that point back to the screen they came from.

What it does

Dockt is a desktop companion that builds searchable memory from screen activity. Its capture agent watches for meaningful screen changes and saves captures with their app, window, and time context. Capture can be paused, and the agent has basic exclusions for private windows and password managers.

Ask Dockt a question in plain language, and it searches the indexed history. Answers include links to the relevant captures, so you can inspect the original screenshot and extracted text. Dockt also gathers detected promises, requests, and deadlines into a linked list; drafts follow-up emails, messages, to-do lists, or notes from selected captures; and can turn activity history into a standup or timesheet.

How we built it

A laptop capture agent sends screenshots and context to Dockt’s Python and FastAPI service. The laptop stores the screenshot files and a SQLite database containing summaries, extracted text, and embeddings. SQLite FTS5 supports keyword search, while NumPy is used for vector similarity; Dockt combines those results and can fall back to keyword-only search if a query embedding request fails.

In the intended local setup, Ollama serves the vision, chat, and embedding models on an ASUS GX10 over the local network. The vision model summarizes screenshots and extracts visible text; the chat model can use Dockt’s search and drafting tools. The screenshot archive and search index stay on the laptop, while screenshots and context travel to the GX10 for model processing.

The project also includes a desktop companion with quick chat, voice input, and spoken replies using local speech components.

The agent: Questions go to a tool-calling agent on the laptop, powered by Nemotron 3 33B on the GX10. Instead of a single "search, then answer" pass, the model decides what to look up. It can call search_screens, get_activity (what was on screen in a time range), get_captures, list_commitments and draft, for up to four rounds before it has to answer. Screenshots cited in earlier answers are reloaded on each turn, so a follow-up can ask the agent to act on what it just found.

Alongside the agent, a background extractor reads each chat, email, or page once it settles on screen, and keeps a running list of promises, requests, and deadlines for each conversation. It runs on the same Nemotron 3 33B, reading the screen text that the vision model has already extracted. Each call also sees the tasks still open in that conversation and decides, for each one, whether to keep it, close it as done, update it, or cancel it. A later "actually, never mind" cancels the original request instead of being ignored or added as a new task. Nothing is deleted; every change keeps the message that caused it.

Challenges

The hard part was making screen memory useful without making it a black box. A generated answer needs to lead back to its source, and a missed or unreachable model should not turn into a confident answer. Dockt links answers and commitments to captures, and its search code keeps keyword retrieval available when query embeddings fail.

We also had to describe the privacy boundary accurately: the archive is stored on the laptop, but a screenshot is sent over the local network to the GX10 when a model processes it.

Accomplishments

The current codebase includes screen capture, local storage, indexing, source-linked chat, commitment tracking, copyable drafts, activity summaries, and a desktop voice companion. It also includes a mock-model path for developing the interface and agent flow without relying on the GX10.

What we learned

A memory assistant is only as useful as the information it captured and indexed. Showing the source screenshot makes answers easier to verify, while separating storage from model inference gives each device a clear role. Local-network AI also brings its own setup and connectivity challenges.

What's next

Next, we'd like to add settings to change capture cooldown, do more experimentation with models for a context window that can last weeks or months, instead of hours, and build a version fully compatible with MacOS.