Inspiration

Everyone has a moment a day where they think "I'm going to forget this": where the spare keys went, the air filter size, the book a friend recommended. Notes apps make you open an app, pick a folder and type into a form, so we never do it. But we always have iMessage open. We wanted remembering something to be as easy as texting a friend, and finding it again to be as easy as asking that friend.

What it does

Pocket is an iMessage contact you text whenever you want to remember something.

Save: "Spare keys are in the green bag in the hall closet" gets a 👍 tapback, sent only after the note is written to disk.
Recall: "Where are my spare keys?" gets "Green bag in the hall closet. You saved that on Oct 10."
Update and history: move things around and Pocket keeps a dated history, so you can ask where something used to be.
Forget: "Forget where I keep my keys" removes it.
No made-up answers: if you never told it, it says "You haven't told me that yet."
Voice: send a voice note to save or ask. Ask by voice and Pocket answers with a voice note, powered by ElevenLabs.
Photos: send a photo of the storage bin. Later ask "Which bin has my tent?" and Pocket answers and sends the photo back.
Actions: "Dentist Tuesday 3pm" saves the memory and adds it to your calendar through Apple Shortcuts. It can also add reminders and notes, start a Focus, or run a HomeKit bedtime scene.

Every memory is a plain Markdown note in your own Obsidian vault, linked to the people and places in it, so Obsidian's graph view becomes a live map of your memory.

How we built it

iMessage: Photon's Spectrum SDK (spectrum-ts) hosts the iMessage line. Pocket uses native features: tapbacks to confirm, the typing indicator while it thinks, and threaded replies.
Brain: one structured-output model call per message classifies the intent (save, recall, update, history, forget, action), extracts the fields, and answers recall questions only from the sender's own notes. It runs on Gemini by default, with Claude as a drop-in alternative behind the same small interface.
Storage: an Obsidian vault on the Mac. One note per thing, with the current value at the top and a dated History section below, plus [[wikilinks]] for people and places. Each sender gets their own folder.
Voice: ElevenLabs Scribe for speech-to-text and ElevenLabs text-to-speech for spoken answers, converted to iMessage's voice-note format with macOS afconvert.
Photos: the model describes each photo into a searchable note, and the image is stored and embedded in the vault.
Actions: a small, allowlisted bridge to Apple Shortcuts for Calendar, Reminders, Notes, Focus and HomeKit.
TypeScript on Node, with Vitest unit tests for the vault, routing, voice, photos and Shortcuts.

Challenges we ran into

Trust. A memory app that is wrong once is useless, so we made two hard rules: the 👍 is sent only after the file is written, and recall answers come only from saved notes, with the date they were saved.
Voice notes in iMessage. Incoming voice notes arrive as CAF audio and outgoing ones need to be M4A with a duration to draw the waveform, so we convert in both directions with afconvert and afinfo.
Photos. iPhone photos arrive as HEIC, which Obsidian can't show, so we convert them with macOS sips. Captions often arrive as a separate text right after the photo, so a short non-question text within 5 seconds is treated as the caption.
Running Shortcuts safely. Letting a text message trigger actions on your Mac is risky. Pocket only runs five fixed Shortcut names from an allowlist, never takes a name from message text, and never uses a shell.
Notes people edit by hand. Because the vault is yours, notes get edited in Obsidian. Parsing had to be tolerant, and updates only touch the current-value line and append to History.
Switching models mid-build. We moved from Claude to Gemini partway through, which pushed us to put the model behind one provider-neutral function.

Accomplishments that we're proud of

It feels like texting a person, not using a bot: tapbacks, typing dots, voice notes back, photos back.
A 👍 always means it was saved, and Pocket would rather say "you haven't told me that yet" than guess.
Your memories are plain Markdown files you own, not rows in someone else's database, and the graph view shows them connecting in real time.
Text, voice, photos and real actions on your Mac all go through one conversation.

What we learned

At personal scale, loading a user's notes into the prompt and telling the model to answer only from them is simpler and more reliable than building semantic search.
Small reliability rules (save before confirming, always cite the date) matter more to trust than clever features.
Native platform details, like voice-note formats, HEIC photos and tapbacks, are most of the work of making something feel native.
Keeping the model behind a narrow interface made it easy to swap providers without touching the rest of the app.

What's next for Pocket

Hosting the agent in the cloud with vault sync, so it doesn't need a Mac running at home.
Semantic search once vaults grow past what fits in a prompt.
Shared memories for households and couples through group chats.
Proactive nudges, like "you parked on level 3" when you head back to the car, or a reminder when a saved warranty is about to expire.
More actions through Shortcuts, chosen by the user.