Familiar: the AI you never open

An AI that lives in the doors you already use - your folders, clipboard, and phone line - backed by one local memory that never leaves your device.

Comment

Inspiration

Every app wants you to open it. Another window, another login, another place to check. But the debris of a real life — receipts, bills, screenshots, missed calls — doesn't live in an app. It piles up in a Downloads folder nobody opens and a phone that rings when you can't answer.

So we asked a different question: what if the AI never had to be opened at all? What if it just inhabited the 40-year-old interfaces you already use — a folder, the clipboard, a phone number — and quietly made your computer know things?

That's Familiar: an AI with no app, backed by one local memory, reachable through doors you already have.

What it does

Familiar runs as a menu-bar helper on macOS. Five doors are live today, and every door reads and writes the same local memory:

  • Sweep — drop a file into a watched folder; Familiar reads it (PDF, DOCX, image), renames Scan_20260721.pdf → Adani electricity receipt — July 2026.pdf, and remembers the amounts, dates, and vendors inside.
  • Paste — one shortcut (Ctrl+Cmd+V) pastes what you meant: messy text becomes a clean TSV table in Sheets, a sharp query in a search bar, a ready-to-send message in chat. Plain Cmd+V is never touched.
  • Lens — Ctrl+Cmd+Space on any on-screen selection to explain, rewrite, or web-research it — inside any app, even VS Code and Chromium, via a universal Cmd+C capture fallback.
  • Ask Familiar — Ctrl+Cmd+K answers questions from everything the doors have learned, grounded in retrieved memory, refusing to fabricate.
  • Pickup — when you don't answer your phone, Familiar answers as your line on a real Indian number, shares only family-tier facts, and refuses anything private.

The money shot: a receipt lands in a folder → Sweep files it silently → an hour later a call comes in → you don't answer → the agent picks up → the caller asks "did he pay the electricity bill?" → it answers correctly, citing the receipt. Nobody told it. Information walked in one door and out another with no human in between.

Core innovation: one memory, many doors

The heart of Familiar is a local SQLite "brain" where every fact carries a permission tier — public / family / private — and every door builds a context bundle filtered to the tier it's allowed to see. When the phone agent asks the brain for context, private rows (a medical report, a bank statement) are physically excluded before the bundle ever leaves the machine — enforced in code, not hidden in the UI. Ask the AI on the phone for a bank balance or a location, and it simply doesn't have it.

How we built it — Codex + GPT-5.6

Every line of Familiar was built with Codex and GPT-5.6, in a single day. Codex wrote the whole stack:

  • a signed native Swift helper driving a global CGEventTap, the macOS Accessibility API, and a screen-capture-invisible preview UI (NSWindow.sharingType = .none);
  • a local SQLite brain with FTS5 search, content-hash dedup, an undo journal, and tier-filtered context bundles;
  • a Vapi voice agent on a bring-your-own Indian DID (+91 22 6973 8970), with the family context injected into the assistant's prompt via the Vapi API.

GPT-5.6 is the runtime brain of every door — it names files, infers paste intent from the focused app, powers Lens, grounds Ask Familiar's answers, and drives the phone agent — all through strict structured outputs with store: false, so nothing is retained or trained on.

Codex didn't just autocomplete — it diagnosed real macOS failures (TCC permission churn from unstable code signatures, AXSelectedText failing inside Electron/Chromium) and fixed them with a byte-stable signing identity and a synthetic-Cmd+C capture path with 250ms clipboard-safe polling.

Challenges

  • macOS permissions. Every rebuild changed the helper's signature, resetting Accessibility/Input-Monitoring grants and silently killing the event tap. Fixed with a stable self-signed identity so permissions survive rebuilds.
  • Reading selected text everywhere. AXSelectedText is empty in Electron and web content. We fell back to the industry-standard synthetic Cmd+C + clipboard-restore, so Lens works in VS Code and Google Sheets.
  • Honest telephony. Vapi's own numbers are US-only; we routed a real Indian DID through a BYO SIP trunk so Pickup answers actual inbound calls.

What we learned

The hard part of an "ambient" AI isn't the model — it's the boundaries. Getting the OS out of the way (permissions, event taps, invisible UI) and getting privacy provably right (tier filtering before the network) mattered more than any prompt.

What's next

  • CC — a door you add to an email thread that tracks commitments and open loops (built against a local inbound-webhook; one config step from live via Cloudflare Email Routing).
  • Kin — consent-first remote help for a parent's Android phone (designed, roadmap).
  • Encrypted device-to-device sync so the same brain follows you across machines.

Built With

Share this project:

Updates