Inspiration

Many older adults want to keep living at home while their families live hours away. The worries never stop: Did Mom take her pills? Is she drinking enough water? Would anyone know if she fell? A phone call can't carry a glass of water across the room.

So we built Miguel. He's knee height and balances on two wheels, so he fits between furniture on less floor than a sheet of paper, and he carries things in a small basket. An iPhone on top, tilted up to see faces, is his eyes, face, voice and ears.

What it does

Miguel is a caretaker robot for a senior at home. She talks to him, and her family reaches him from an app or by text.

  • Listens. "Miguel, bring me some water" or "follow me", in her own words. "Stop" works instantly, without waiting on the AI.
  • Delivers. Mom asks, or the family picks Water or Medication in the app and adds a note. Miguel says "On it!", gets loaded at his station, brings it over and reads her the note word for word.
  • Texts and talks with the family. On iMessage, "how's Mom?" gets a two-line answer from her day, and "bring her water" starts a delivery that ends with a photo. They can call him too.
  • Finds and follows. "This is Mom" teaches him what she's wearing. He turns to look for her, and follows one person while ignoring everyone else.
  • Watches for falls. If he sees someone on the floor, the AI double-checks a short clip. He texts the family a photo, asks if she's okay and passes on her answer. If no one replies within a minute, he calls.
  • Checks in. The family's video call opens only after she says yes out loud.
  • Remembers places from printed signs, the people he's met, and a log of Mom's day.

He announces every move, never assumes who's talking, and tells the family when something goes wrong instead of guessing.

How we built it

  • Body: an ESP32-S3 runs a 200 Hz balance loop on an MPU-6050 IMU and two NEMA 17 steppers. The 3D-printed body weighs 2.2 kg and holds an iPhone on a 24 cm mast.
  • Brain: Qwen3.8 27B on the laptop (through Ollama) picks skills like find_person, go_to and bring, and one setting swaps in Claude Sonnet 5.5. It never drives the wheels directly. A SQLite memory and a note on who's in view go into every request.
  • Eyes: the iPhone's wide front camera streams to the laptop, where Ultralytics YOLO pose finds people, a lock on clothing colors keeps the right one, and printed ArUco signs mark places.
  • Voice and face: Parakeet TDT 0.6B v3 with Silero VAD hears her and Kokoro-82M speaks, all on the laptop, so her voice never goes to the cloud. On an M5 Pro, Qwen's first sentence comes in about 2.8 s and a spoken line takes 0.07 s. His "Mochi" face follows you with its eyes and moves its mouth as he talks.
  • Safety: one arbiter decides who drives: the emergency stop, then a person's joystick, then skills. On the robot, a 300 ms watchdog and a 30° tilt cutoff work even if the laptop fails, and the server won't start him until he's upright and his tilt sensor is steady.
  • Family: a Next.js web app with a 3D Miguel in three.js, and Photon 2.0 for iMessage and calls. Every channel feeds the same agent and memory.

We wrote all the firmware and software during the hackathon, each of us with an AI coding agent: Claude Code (Claude Opus 5.5) wrote most of the code with us, and Codex reviewed it.

Challenges we ran into

Balancing with a phone on a stick. Tilting the phone up moved the balance point by 7°, and taking the kickstand off moved it again. We re-measure after every change, and the control page adjusts it live.

Safe testing. After a follow run, Miguel lost his balance, and the firmware's tilt cutoff stopped him on its own. So safety lives on the robot, where no laptop or network failure can turn it off. When his tilt sensor started drifting, we taught the server to catch it.

The right person from knee height. The camera saw mostly legs, and the nearest bystander kept stealing the lock. Tilting the camera up, locking on clothing colors and judging distance two ways fixed it.

Pauses feel broken. Cloud models took 2–4.5 s to say anything real. An instant "On it!" fixed how the wait felt, and Qwen on the laptop answers follow-ups in 0.7–2.2 s.

Venue internet. The guest Wi-Fi needs a sign-in page a robot can't complete, and our voice account ran out of credit. So we run our own hotspot, and the brain, hearing and voice all run on the laptop.

Accomplishments that we're proud of

  • He really balances: hands-off within about 1°, with an iPhone on a 24 cm mast, over about 75 m of hallway.
  • Plain words become plans. "Bring Mom her pills" turns into steps he carries out, says out loud and reports to the family.
  • Consent in code. The video call opens only after the mic hears her say yes. The AI's word isn't enough.
  • No cloud needed. A 27B open model on a laptop plans his tasks, reads the camera and checks for falls.
  • The same Miguel, however you reach him. Mom talking to him, the family's app, texts and calls all go to one brain with one memory. If her son texts "bring her water", her daughter can call an hour later and Miguel already knows it was delivered.
  • Four lanes, one robot, built in parallel against a mock robot, with over 800 automated tests.

What we learned

  • A balancing robot is a center-of-mass problem. Every gram on the mast matters.
  • Put safety where software can't break it.
  • Let the AI choose, not drive. Skills instead of motor commands made Miguel smarter and safer.
  • Prompts guide; code guarantees. The rules that protect her live in code.
  • For voice, how fast it feels beats how fast it is.
  • Plan for the venue, not the lab. Every cloud piece got a local fallback.

What's next for Miguel.exe

  • Modules: today he's a mast with a phone and a basket. Next come printed attachments a family can clip on, like a gripper or a pill box.
  • A taller Miguel, around waist height, so Mom never has to bend down.
  • Routines: her pills on schedule, without anyone asking.
  • Knowing Mom by her face, with her consent.
  • The whole home, with more stations and the iPhone's LiDAR.
  • A pilot with real families, with the senior always in control.

The tracks we're entering

  • Best Healthcare Hack. Miguel helps a senior keep living at home. He brings water and pills when she asks, reads her family's messages word for word, sends fall alerts with a photo, and opens a video check-in only after she says yes. Her voice never leaves the laptop, and one word stops him.
  • Best Hardware Hack. He's a 2.2 kg two-wheeled balancing robot with an iPhone on a 24 cm mast. He holds within about 1° on a 200 Hz ESP32-S3 loop we wrote and tuned here, and his safety lives in firmware: a 300 ms watchdog and a 30° tilt cutoff.
  • Best Physical AI Hack. An LLM runs a real body without touching the motors. It turns plain requests into physical skills, checks a clip before a fall alert, and recovers out loud when something fails. One arbiter, with the emergency stop on top, decides what reaches the wheels.
  • Best Practical AI Innovation (Amazon). An agentic accessibility tool for someone who would never use a screen. Mom just talks, and Miguel carries out multi-step tasks and reports each step to her family. His brain runs on a laptop, so a task costs nothing extra.
  • Photon: Messaging Agent. The family texts Miguel like a person, and a real robot acts on it, with a photo when it's done. A possible fall goes to everyone by text, the first "I've got it" tells the others, and if no one answers within a minute, he calls.
  • Photon: Best Voice Agent. Miguel talks at home and on the phone. At home he has a wake word, speech-to-text with turn detection on the laptop, and a spoken "stop" that works instantly. On the phone, a call ends with a text summary. Whatever is said out loud, in the app or by text goes into one memory.

Built With

Share this project:

Updates

Submission history