Savepoint

Reliving your memories

Comment 1

Inspiration

You've had this moment: someone waves, you smile back, and you have no idea what their name is. Awkward. Forgettable. You move on.

Now watch a parent live that every day, except it never moves on. The person who once knew your school schedule, your friends' names, the exact way you take your coffee, starts asking the same question twice in the same conversation. Starts pausing before your name, just slightly, just long enough for you to feel it in your chest. Nothing about them has changed. The love in their eyes hasn't changed. But somewhere between them and you, the thread connecting today to yesterday has started to fray, and there's nothing either of you can do to stop it.

For people living with short-term memory conditions, that fraying thread isn't occasional. It's constant. And they fight it the only way they can: on paper. Notebooks by the bed. Notebooks in the car. Notebooks stacked on every counter, each page holding a name, a conversation, a promise they made and can't afford to lose. Eventually the notebooks take over a room. Then the house. And eventually, because no house can hold every page forever, they have to start throwing some away. Not clutter. Not junk. Actual days of someone's life, actual people they loved, gone because there was nowhere left to keep them.

What it does

SavePoint is a cozy, gamified journal of your real life. A small Raspberry Pi 5 wearable sees the people you talk to, your phone hears the conversation, and together they turn each person you meet into a deterministic pixel character, each day into a plant in a garden, and each conversation into dialogue boxes you can replay.

Talk to someone at a party, a conference, a family dinner. Within a few seconds they appear on screen as their own sprite. Their words attach to them, journalized and time-stamped. At the end of the day, "Today" is waiting for you: a cinematic day-view with a timeline scrubber, a garden that's grown one more plant, and a short, warm recap of who you saw and what you talked about.

We built it game-first. The primary hook is charm, not therapy, a low-pressure record of your social day that happens to double as a gentle and easily useable memory aid. It's also privacy-first by construction: only derived data ever leaves a device (sprite parameters, voice embeddings, transcript text), never raw faces or photos, and even a hardware mute switch on the wearable can physically cut the camera.

How we built it

Edge: A Pi 5 on regular Raspberry Pi OS (no RTOS) is its eyes. A hardware abstraction layer sits behind two backends: a simulation backend, fully tested in CI, and a linux backend built for the real device (picamera2, gpiozero for a GPIO mute button + LED, onnxruntime for optional on-device face detection). Only derived events, sprite parameters and embeddings, ever leave the device.

Speech: An existing speech pipeline figures out who's speaking and transcribes their words into Speaker N: text events, behind a Transcriber protocol, a stub for CI, the real pipeline via subprocess.

Backend: FastAPI + uv, with MongoDB as the actual game world, not a bolt-on. Four collections map 1:1 to the product: people (avatar params, voice embedding, tags), events (who said what, when), days (a computed plant-growth signal), and recaps (generated narratives). A single write path, POST /ingest, takes a frame and audio, matches or creates a person, appends events, and rolls up the day. A full read API, /today, /day/{date}, /people, /days, composes those documents into everything the app renders.

AI: Recaps and bios run through one pluggable LLMClient, swapped by a single config value. It's live today on a self-hosted Gemma endpoint, and the same seam is what lets Gemini and FreeSolo slot in without touching a call site.

App. React 19 + HeroUI v3 + Tailwind v4 + framer-motion, a PWA built as a cozy pixel game: a character plaza, a calendar garden, and a cinematic Day view with dialogue boxes and a scrubbable timeline, all wired to the live backend.

Challenges we ran into

The data model is a contract stretched across four independent workstreams, edge, speech, backend, app, so we made the server's Pydantic models the single source of truth and kept the app's types deliberately shaped to match, so wiring UI to live data stayed mechanical instead of a guessing game.

Binding "who spoke" to "who's on screen" turned out to be genuinely hard: the mic lives on the phone, the camera lives on the Pi, two separate clocks. We decided the server aligns both streams by timestamp, and kept tap-to-assign as a demo fallback if the alignment gets loose on stage.

We wanted a very specific voice out of our recaps, cozy, Stardew-toned, never clinical, without the model hallucinating people who were never in the room. We solved that by constraining prompts tightly to a day's actual events and capping the number of highlights, so the narrative stays grounded in real transcript data.

Building an edge device you can't always have plugged in meant we couldn't let hardware availability block the team, so the HAL plus simulation backend let the whole capture pipeline get built and CI-tested with zero hardware in the loop. And a spike into FreeSolo's Flash service taught us mid-hackathon that it's a fine-tuning service, not a drop-in API, which meant re-scoping that track from "swap a URL" to "train and deploy an adapter" without slowing down the recaps we already had shipping on Gemma.

Accomplishments that we're proud of

The whole core loop, person seen, character created, words attached, day recapped, is live end-to-end on real data, not a mockup. MongoDB isn't a database bolted onto a demo; it's the literal game world the app renders. Every person becomes a genuinely deterministic sprite: the same face produces the same character every time.

We're proud that privacy isn't a settings toggle here, it's architectural. Raw video and audio never leave the device boundary we control; only derived parameters do, and the mute switch is a physical guarantee, not a promise in a menu. And we built a generation architecture, one pluggable LLM interface, that turned what could've been five separate prize integrations into one clean seam with five backends behind it.

What we learned

Design the pluggable seam before you need it. Building LLMClient as a swappable interface early, before we knew which model backend would actually win the demo, is the single decision that let us chase four different AI prize tracks without rewriting anything.

Simulate the hardware you don't have in hand yet. A HAL with a tested simulation backend meant the team never sat around waiting for a Pi to be free, and it forced us to design the real interface (camera, mute, detector) cleanly instead of writing code around whatever the hardware happened to expose.

Read the fine print on a sponsor's API before you commit a track to it. Assuming FreeSolo was a standard hosted endpoint cost us a day we got back by spiking early instead of late. And dropping our original QNX plan for plain Raspberry Pi OS was its own lesson: a more exotic platform is only worth it if the ecosystem underneath it actually supports what you're building, camera drivers included.

Privacy-by-construction is cheaper early than late. Deciding upfront that only derived data leaves the device meant we never had to retrofit a "trust us" story onto raw video we were already collecting.

What's next for Savepoint

We want to grow the world past the fixed garden grid into an expansive world that traces and builds itself around the user live, so the places you actually go become part of the map, and the people you meet show up at the real spot where you met them. This can further help dementia patients identify the specific locations as well as the people they met there, helping them rebuild what they are unable to do so themselves.

Built With

Share this project:

Updates

posted an update —

MongoDB Atlas:

  • Accessed through Motor (AsyncIOMotorClient), so every DB call is async/await and non-blocking under FastAPI
  • Single process-wide client, created lazily on first use
    • Importing the app (CI, unit tests, type-checks) never requires a live database
  • Connection configured via pydantic-settings

Collections:

  • people - one document per person you've met: local_id, avatar_params (6 sprite axes), face_embedding (512-d), voice_embedding, tags, favorite, first/last-seen
  • events - an append-only interaction log; each row is a person seen or a line spoke, carrying ts, person_id, day_id, text

Deterministic identity:

  • People are keyed by local_id (the stable, edge-assigned identity from the Pi)
    • The same face → the same _id → one document, forever
    • Re-seeing someone is an upsert-by-key (replace_one(..., upsert=True)), not a duplicate
  • Days keyed by ISO date (2026-07-19), recaps keyed by ":", wearer-voice keyed by the literal "you"
  • Events are append-only with a random uuid4 _id

Architecture:

  • A typed repository layer wraps Motor: every method returns Pydantic models, never raw BSON, with _id ↔ id mapping isolated in one base class
  • All four workstreams evolve the shape together without migrations.
  • Privacy by construction: only derived data is ever persisted
    • Sprite params, embeddings, transcript text
    • No raw photos, video, or audio ever touch the database for security

Log in or sign up for Devpost to join the conversation.