Usme

• 

• 

• 

• 

• 

• 

Inspiration

Everyone already has a memory system. Notes apps, voice memos, screenshots, calendar invites, links saved for later. The capture problem is solved.

The recall problem is not.

Six weeks after you write something down, you can't find it — because you search for it in the words you're thinking now, and you wrote it in the words you were thinking then. You ask "what did my doctor say about my cholesterol?" and the note reads "Dr. Yılmaz — LDL 164, cut the red meat." Not one word in common. Keyword search can't bridge that gap, which is why the note might as well not exist.

And the failure that actually costs you something isn't search at all. It's the promise you made in a meeting on Tuesday, fully intending to keep, forgot by Thursday, and heard about on Monday.

I wanted an app where the answer to "where did I put that thing I knew?" is a question, not a folder.

What it does

Usme captures what you experience — voice, text, calendar events, the iOS share sheet, photos — and reads each one for the people, projects, places and promises inside it. Those entities are resolved against each other, so "John", "John Carter" and "JC" become one person, and linked into a graph you can walk.

Then you ask it things, in your own words, and it answers with citations you can tap through to the memory the answer came from.

And when a deadline you spoke aloud is approaching, it tells you before the other person has to.

It lives where you already work. Three App Intents — Save to Usme, Ask Usme, and Record a Memory — put the whole app into Shortcuts, Siri and Spotlight, with ten spoken phrases. Save and Ask run in the background, so a shortcut ending in "save this" never drags you out of what you were doing, and Ask returns its answer as a value you can chain into any other action. A Home Screen widget, a Lock Screen accessory and a Control Centre control — assignable to the Action Button — each open straight into a running recording. A widget cannot hold a microphone, so rather than pretend, they remove every step between the thought and the recording.

Every answer's citations open. Tapping one shows the memory, the people and projects read out of it, the promises extracted from it, and what it connects to with the shared names spelled out. It is also the only place a memory can be corrected — on-device transcription mishears names, and one wrong word otherwise propagates into an entity and a commitment with no way back.

And the data can leave. Export writes every memory, person and promise to plain JSON with the attachments beside it, because local-first has to mean the data can get out or it just means the data is trapped. Memories are indexed into the iPhone's own search too, behind a switch on the Privacy screen — it is the one thing Usme does that makes a memory legible outside the app, so it is named rather than assumed.

How I built it

Local-first, genuinely. SQLite on the device is the source of truth, not a cache. Capture, storage, search, ranking and reminders all work with no network at all. Speech transcription and OCR run on-device — audio and photos never leave the phone.

Retrieval is the product, so it got the engineering. Three arms run independently:

1. Keyword — SQLite FTS5 with BM25, over a Porter-stemmed, diacritic-folded index
2. Semantic — brute-force cosine over normalised embeddings via Accelerate's vDSP_dotpr. No vector extension, no index to corrupt; at this corpus size it's sub-millisecond
3. Graph — when a question resolves to a known entity, expand one hop and boost what's connected

They're fused with Reciprocal Rank Fusion, which combines ranks rather than scores. That's the whole reason it's there: BM25 is unbounded and cosine is bounded to [-1, 1], and every naive hybrid search dies trying to calibrate one against the other. Ranks need no calibration.

Ahead of retrieval, a fast model rewrites the question into the vocabulary the memory probably used. That rewrite is what makes the doctor question work.

The backend is a proxy, not a backend. ~550 lines of TypeScript on Cloudflare Workers. It exists for two reasons that can't be solved client-side: an API key shipped in a binary is an API key on the internet, and a free-tier limit enforced on the client is a limit enforced by nobody.

Monetisation via RevenueCat, with the entitlement check on the server. The app sends RevenueCat's anonymous app user ID, and the Worker uses the same ID for both the entitlement lookup and its quota counters — so a subscriber and their usage are the same person, and the limits can't be patched out of the client. It fails closed: a misconfiguration costs me revenue, never a free ride. No accounts, no login, no account-deletion requirement.

Stack: Swift 6.2 / SwiftUI (iOS 18+), GRDB, Claude Haiku 4.5 for extraction and query planning, Claude Opus 5 for answers, Voyage for embeddings, Cloudflare Workers + KV, RevenueCat. 64 Swift files, ~10,400 lines, 125 tests.

What I learned

Recall quality is invisible without a gate, so I built one early. 37 seeded memories, 25 questions with known-correct sources, asserting recall@5. Every retrieval change runs against it.

• Lexical questions — 100% either way
• Entity questions — 100% either way
• Semantic questions — 33% with mocks, 78% with real providers
• Overall — 76% with mocks, 92% with real providers

That harness taught me the most important lesson of the project, by lying to me.

An early run reported semantic recall at 33% with real providers — identical to the mock. The obvious conclusion was that real embeddings changed nothing. Then the diagnostic said five expected memories were "not in the top 50" of a 37-memory corpus, which is arithmetically impossible.

The real cause: Voyage's free tier throttled to about three requests a minute, most query embeddings were being rejected, and a single try? in the retrieval path was swallowing every failure. The semantic arm wasn't underperforming. It was absent, and nothing said so. Logging the failure instead of swallowing it, batching the calls, and adding a preflight that fails the suite rather than publishing a degraded number took semantic recall from 33% to 78%.

That turned out to be the theme of the whole build: almost every serious bug was silent.

• The ingestion queue ordered jobs by a string column, and "embed" sorts before "extract" — so the pipeline ran backwards, embedding memories before anything had read them. No error. Found only because 37 sample memories loaded with zero entities.
• Notifications were "delivered" while the app was in the foreground, where iOS silently drops them — and then marked as sent, so those promises would never have been mentioned again. My first fix made it worse: it skipped foreground delivery entirely, leaving only background tasks, which iOS almost never grants a freshly installed app. The real answer was architectural — a reminder for Friday has to be scheduled for Friday, so it arrives whether the app ever runs again or not.
• A due date with no UTC offset was rejected by the parser and came back nil. The deadline wasn't wrong, it was gone, and the promise never nudged.
• Building with CODE_SIGNING_ALLOWED=NO silently strips entitlements, so the App Group fell back to a private container and the share extension's handoff quietly stopped working.

I now reach for "what would this look like if it were broken?" before "does this work?"

Challenges

Two crashes on real hardware, both the same mistake wearing different clothes. The first was an explicit MainActor.assumeIsolated inside a speech callback that runs on an arbitrary queue. I fixed it — and it crashed again, this time because the audio tap closure was written inside a @MainActor type and implicitly inherited that isolation, so Swift emitted an executor check on every audio buffer and dispatch_assert_queue trapped on the realtime thread. The crash log named it exactly. Fixing the explicit case hadn't found the implicit one, and the same latent bug was sitting in the iOS 26 code path, uncrashed only because no one had recorded on that OS yet.

My own phone forced a rewrite. I develop on an iPhone XR, which tops out at iOS 18 — and I'd built voice capture on iOS 26's SpeechAnalyzer. I literally could not run my own app. Rather than ship a hero feature that had never executed on hardware, I put transcription behind a protocol with two implementations. The awkward part: @Observable expands over stored properties, and a property typed SpeechAnalyzer? can't be version-gated in place — the gate has to be a whole type.

Long recordings lost everything after a minute, because SFSpeechRecognizer finalises a task after roughly sixty seconds and then reports nothing. Segments are now chained, each finalised one banked while the audio engine keeps running.

"Meeting at 2pm today" became 5pm. Exactly my UTC offset — which is the tell. The app was sending the model a UTC timestamp, so it reasoned entirely in UTC and answered 14:00Z. The model was never told my timezone existed.

**The Control Centre control shipped broken, and looked fine. It used the system's OpenURLIntent to reuse the URL the widgets open — it builds, it passes review, it appears in the gallery, and at the tap it fails with "The provided URL scheme usme is unsupported; launch is prohibited". OpenURLIntent will not launch a custom scheme; widgetURL still can, which is why the widgets worked and the control did not. I had verified that the control appeared, which is not the same as verifying that it works — the same lesson as the retrieval eval, learned again.

And the name. The app was called MemoryOS until I found an established company using that name with the matching .com. Eight candidates later — every evocative single word in this category is taken, usually by something shipped in the last twelve months — it became Usme: the initials of me, my wife, and my son.

What's next

Sync across devices, a Core ML embedding model to bring semantic search fully on-device, and connectors for the places the rest of a life already lives.

Built With

• swift
• typescript