Privacy first.
Ask the app, and hear your loved ones tell the story.
Record the voices of the people you love.

Inspiration

Everyone has someone whose stories they wish they'd recorded. A grandparent who told the same story about the diner on 4th street every holiday. A dad whose road-trip tales got better every year. When those people are gone, photos stay, but the voice, the pauses, the laugh in the middle of a sentence, all of that disappears first.

Recording those stories is only half of it. We kept coming back to a bigger question: what if you could still ask them? Not a chatbot pretending to be your grandmother, but a way to ask your family's memories a question and hear the answer in the moment they actually said it, in their own voice. That's Echoes.

What it does

Echoes helps you capture, understand, and relive the stories of the people you love.

Guided storytelling: a deck of prompts ("Where did you go on your first date?") helps anyone start talking.
Record and save instantly: stories are saved right away and processed in the background, so nobody waits on a loading screen.
Accurate on-device transcripts: Whisper large-v3 runs on the iPhone itself, with Apple's SpeechAnalyzer as a fast fallback. During playback, the transcript highlights each line as it's spoken, and you can tap any line to jump there.
Joy pins: the front camera notices when the storyteller smiles and marks those moments on the timeline.
Understands each story on-device: Apple's Foundation Models write a title, summary, themes, people, places, and years, and suggest follow-up questions to ask next time.
Talk to your memories: ask "How did Grandpa meet Grandma?" by voice or text. Echoes finds the right passages, answers using only what was actually said, reads the answer aloud, and then plays the original recording of them telling it.

Privacy is the foundation. Recordings never leave the phone. Talk is opt-in and needs three yeses before any words are shared: the app setting, the memory itself, and the consent of the person who told the story.

How we built it

iOS app: SwiftUI and SwiftData, with a versioned schema so existing memories migrate safely between versions (V1 → V2 → V3).
Transcription: WhisperKit (Whisper large-v3 on the Neural Engine) and Apple SpeechAnalyzer behind one provider interface, producing timestamped segments that drive the synced transcript and the clips.
On-device intelligence: Apple Foundation Models for enrichment and smart search, and the Vision framework for smile detection.
Backend: a stateless Cloudflare Worker in TypeScript that keeps every API key server-side. It uses OpenAI (gpt-4.1-mini with structured output) for grounded answers with citations, and ElevenLabs for narration. Rate limits, a constant-time auth check, no database, and no logging of what people ask.
Team: we split into three tracks (data and transcription, on-device processing and search, backend and Talk), agreed on a shared data contract first, and built in parallel.

Challenges we ran into

Choosing a transcription engine. We benchmarked WhisperKit, Apple's SpeechAnalyzer, and OpenAI's API. OpenAI's most accurate model returns no timestamps, and without timestamps we couldn't sync the transcript or play the exact clip. On-device Whisper won on accuracy, with Apple's engine as the fast fallback.
Search that actually finds the story. Sentence embeddings found the right memory for only 3–5 of 10 test questions. AI-generated keywords plus on-device smart search did much better.
A deploy that silently failed. Every provider call failed on Cloudflare because Workers rejects a fetch redirect setting our mocked tests never exercised. We tracked it down with local runs and fake keys, then added a regression test.
Memory on a phone. A 600 MB Whisper model plus the camera is a lot. We made sure two models can never be loaded at once, even when a new recording starts while the last one is still being transcribed.
Never losing a memory. Every change to the data model has to carry existing stories forward, so we tested each migration against real stored databases, not just fresh installs.

Accomplishments that we're proud of

Asking a question and hearing the answer, followed by the storyteller's real voice saying it, on a real iPhone.
Answers that say "I don't have enough information" instead of making things up.
Transcription, summaries, themes, and search all run on the device; the cloud is used only where it truly helps, and only with consent.
Three people, three tracks, one shared data model, merged without breaking each other's work.

What we learned

Timestamps matter as much as accuracy: they're what turn a transcript into a way back to the moment itself.
On-device AI is now good enough to do most of the work privately.
Consent has to be designed into the product from the start, especially when the voices belong to other people.
Mocks hide real-world problems; test against the real thing early.

What's next for Echoes

Faster Talk: streamed narration, live transcription while you speak, and playing the original clip during the wait.
Family sharing: let relatives contribute stories to one shared, private family archive.
More languages, so grandparents can tell stories in the language they think in.
TestFlight for families to try it, followed by the App Store.