ReMind
The physical robot arm that shuffles the blocks for the "Remember the Order" game
The home screen
The nurse / caretaker's dashboard

Inspiration

Nearly half of nursing home residents live with dementia, and the activities that help them, like sorting familiar objects or finishing a proverb, need a staff member sitting next to them. Staff are stretched thin, so many residents spend much of the day doing nothing. Several of us have a family member in this situation. Existing digital tools tend to be fixed, hard for older adults to use, easy to abandon, and thinly evidenced, so we built a memory-game companion you talk to instead of tap, with a real robot arm so the resident does something with their hands.

What it does

ReMind is a voice companion that a caregiver sets up and supervises. The resident talks, a calm screen shows one photo or a few words, and the caregiver reads a plain summary afterwards.

The main game is "Remember the Order". The resident says something like "let's play the shuffle game," and ReMind starts it on its own: the robot arm shuffles three keychains on the table in front of them, then waits. The resident is free to put them back in order by hand, at their own pace.

ReMind also has three conversation games based on Cognitive Stimulation Therapy: famous faces, finishing a saying, and everyday sounds. If the resident says something worrying, like a fall or chest pain, Gemini decides whether it matters, reassures them calmly, and texts the caregiver a short summary using Photon. After each session the caregiver sees plain notes about how it went, never a score. The AI can't move the robot on its own. It can only ask from a short list, and our server and the robot's software both check every request.

How we built it

ReMind uses a Python/FastAPI backend and a TypeScript/Next.js frontend. Voice runs on Gemini's Live API through Backboard, which keeps long-term memory, and our backend relays the audio so the API key stays off the browser. Caregiver alerts go through Photon: its SDK is TypeScript, so a small Node service turns a request from our backend into an iMessage.

The robot is an SO-101 arm with two cameras, run through LeRobot. We taught it by demonstration: one person moves a "leader" arm and a second "follower" arm copies, while the cameras film each 35-second try. We recorded 60 tries, 12 for each of five shuffles, and trained policies on Perceptron's cloud platform. A robot worker re-checks every job against its own allowlist and joint limits before the arm moves.

Challenges we ran into

The biggest challenge was getting the arm to grasp. Our trained policies move smoothly and stay inside safe limits, but after 1,000 training steps they can't pick up the keychains reliably. So the demo arm runs taught positions with the camera verifying the result. We verified two shuffles, swap left/right and swap left/middle, in both directions. The other three stall at the empty fourth slot, which the camera can't see. Re-picks there worked 5 times out of 7.

The robot is also picky about its surroundings. Table color, lighting and camera angle have to match between recording and use, and even a keychain placed slightly off by hand can make it miss, so we taped everything down and set up the table with a script where the arm points to each slot.

On the voice side, default voice detection decides you're done speaking too quickly, which cuts off people who pause mid-thought. We waited 1.2 seconds of silence before answering. A shuffle also takes many seconds, so game start returns immediately and the AI speaks up when the arm finishes. Photon's free tier only texts enrolled phones, and its error didn't say so. We spent an hour blaming our own code before we tried Photon's starter project, which failed too.

Accomplishments that we're proud of

Safety enforced in code: The AI can only call a short list of tools, and the rules live in our server and the robot software, not in the prompt, so the AI can't talk its way past them.
Camera-verified robot moves: Every shuffle is confirmed by the camera, with an honest account of what works and what doesn't.
An end-to-end alert: A worrying sentence becomes a calm reply for the resident and a short text for the caregiver, without alarming the resident.
A game with no scores and no "wrong": It's built on errorless learning and Cognitive Stimulation Therapy principles.

What we learned

We learned that keeping the AI behind a short, validated tool list is what makes it safe enough to connect to a robot. We also learned that real-time voice is mostly about when to stay quiet, and that for vulnerable users a kind tone and no scores matter as much as accuracy.

Robot demos fail on the environment more often than on the model, so the setup is part of the product. A scripted fallback with a camera check beat an unreliable learned policy for a demo, and it's fine to say so.

What's next for ReMind

A learned policy that grasps on its own: more training steps and more demonstrations.
A camera that can see the fourth slot, so the other three shuffles become reliable.
More skills for the arm: each new policy we teach it could become a new hands-on game.
Games that adapt to each person: today we pick games per person, but we don't adjust difficulty yet.
A pilot with a care facility and an occupational therapist, before we claim any benefit.

These steps would turn ReMind from a prototype into a companion a care home could try with real residents.

Code: github.com/remind-dubhacks/dubhacks-2026