watch me if you can

• 

• 

Inspiration

i work across research, code, conversations, documents, meetings, and multiple projects in the same day. each tool holds one part of the work, but none of them remembers the complete context: what i was trying to do, what i had already checked, why something mattered, or what should happen next.

the more work i do in parallel, the more valuable that continuity becomes. so i built watch-me as a persistent context layer for the entire computer. it turns visible work into structured, private memory so the context behind an app, message, file, or decision remains available after the window disappears.

i also did not want another separate agent harness running on my computer. i already use codex to get work done, so i designed watch-me around the codex engineering loop itself. watch-me provides the private memory and perception layer, while codex becomes the reasoning layer that searches, connects, and acts on that context.

What it does

watch-me observes only changed foreground windows. it rejects excluded apps, private browser windows, and secure fields before capture, removes sensitive information with local OCR, and uses moondream 2 on the mac to turn each useful frame into a structured observation.

those observations become an encrypted timeline that i can search through codex using normal questions like “where did i see that error?” or “what was i working on before this meeting?” codex can recover the matching time, app, topic, observation, and retained redacted frame.

watch-me can also run focus sessions. i give it a concrete goal, such as finishing a launch note, and it creates a dedicated codex coworker task that retains that exact intention. moondream labels changed activity as relevant, neutral, distracting, or uncertain.

one unrelated page is not enough to trigger anything. watch-me requires at least three high-confidence distracting observations sustained across 90 seconds. when that threshold is crossed, it resumes the same codex task, restores the original goal, and gives me one useful action that takes under five minutes.

the system remains advisory. it never blocks websites, closes apps, types for me, or sends screenshots to codex. i can correct mistaken classifications, customize how codex should recover me, pause observation instantly, or run memory-only mode without focus nudges.

How we built it

watch-me is a uv-managed python application for macOS. pyobjc handles foreground-window capture and accessibility context, while macOS Vision performs local OCR redaction. sensitive regions are removed before moondream or storage sees the frame.

moondream 2 runs entirely on the mac through a separate model worker. changed-frame detection, bounded newest-frame queues, offline model loading, and restart recovery keep local inference responsive and reliable.

captions, visible text, embeddings, indexes, focus goals, corrections, and retained frames are encrypted with AES-GCM. the key remains in macOS Keychain. observations are stored in an encrypted append-only ledger with exact and semantic search.

an authenticated local MCP server gives codex narrow tools for searching memory, retrieving evidence, controlling privacy, and managing focus sessions. a menu-bar app provides simple controls, while a dormant LaunchAgent gives the watcher a stable macOS permission identity without running until the user explicitly starts it.

Challenges we ran into

the hardest part was making continuous observation useful without making it invasive. privacy could not be added after capture, so exclusions, secure-field checks, redaction, encryption, retention, pause, and two-stage deletion had to be part of the architecture from the beginning.

macOS permissions were another challenge because screen recording and accessibility depend on a stable application identity. we had to make the installed app, daemon, and model worker behave consistently across restarts without silently starting capture.

local vision introduced latency, memory pressure, incomplete captions, blank frames, and worker failures. we addressed these with changed-frame detection, bounded queues, strict output parsing, offline model caching, fail-closed behavior, and restart recovery.

focus detection was also more subtle than expected. a short break, uncertain caption, meeting, or necessary research should not become an interruption. this led to the sustained-drift threshold, meeting suppression, cooldowns, user corrections, and the separation between model suggestions and deterministic local decisions.

How we used codex

i used codex as both the engineering partner that built watch-me and the reasoning backend that makes its memory useful.

during development, codex worked as an autonomous improvement loop. after i described a goal or something that felt wrong, it inspected the repository, wrote a test for the expected behavior, ran the suite, implemented the smallest fix, and verified everything again.

when unit tests were not enough, codex operated the real macOS app, inspected actual frames, logs, encrypted records, permission flows, model output, and focus sessions, then used that evidence for the next iteration. real failures became permanent regression tests instead of one-off fixes.

inside the product, codex searches the encrypted memory through MCP, reconstructs working context, answers questions, and restores the exact original goal after sustained distraction. moondream handles private local visual understanding, deterministic code owns privacy and safety, and codex handles interpretation and continuation.

Accomplishments that we're proud of

i am most proud that the complete loop works on a real mac rather than only inside a mocked demo. watch-me captures a real foreground window, redacts it, understands it locally, encrypts the observation, retrieves it through codex, and recovers the original goal after sustained distraction.

we proved the focus flow using real applications and a real codex coworker task. we also proved memory retrieval by recovering an exact observation with its time, app, domain, topic, observation id, and retained redacted frame.

the project passes 82 automated tests covering privacy failures, encryption, search, deletion, permissions, worker recovery, focus thresholds, user corrections, and codex routing.

What we learned

i learned that useful memory is not about capturing everything. it is about deciding what should never be captured, what changed enough to matter, and what context will remain useful later.

i also learned that local models do not need to understand an entire life perfectly. they only need to produce grounded observations inside a constrained system. deterministic code can own privacy and consequential decisions, while codex handles interpretation, reconstruction, and judgment.

finally, focus support works better when it restores context instead of punishing distraction. the useful intervention is not blocking an app; it is reminding me why the original work mattered and making the next step easy.

What's next for watch-me if you can

next, i want watch-me to understand longer project arcs instead of only individual observations. it should connect related work across applications, generate automatic handoffs when i switch projects, surface unfinished decisions, and help codex resume with the right files, conversations, and next steps already available.

i also want to add hotkey-based voice control so i can talk to the background coworker without opening another interface. the longer-term goal is a private, live companion that lives on the computer, remembers the work, fetches context when asked, and helps without constantly demanding attention.

after that, i want to improve visual grounding, support more local models, make privacy controls easier to inspect, and bring the same continuity layer beyond macOS.

Built With

• codex
• gpt5.6
• macos-vision
• moondream
• python
• quartz
• uv