Hey Andy

A small robot that reads the letter in your hand and looks for your glasses: a companion for older adults who live at home, and the family who help. Built with NVIDIA Nemotron on Nebius, and Tavily.

Comment

For the judges

Andy is a small home robot for older adults who live at home, and the family who help. Scams reach people there, by letter and by phone: adults 60 and over reported about $3 billion lost to fraud in 2025 (FTC). Andy reads the letter in your hand and puts Be careful first on a risky one, looks around the room and points at your glasses, and looks things up with Tavily while it keeps talking with you. Measured: 58 of 60 scams caught, none of 60 ordinary messages flagged; 94.8 of 97 spoken situations handled right on average over nine text-only runs, judged by NVIDIA Nemotron 3 Ultra with Andy's real consent check around stand-in tools; both evaluations ran as Nebius Serverless Jobs. Entered in the Physical AI track.

  • Track fit: an embodied agent on a Pollen Reachy Mini. Pollen's conversation app is the agent runtime; NVIDIA Nemotron on Nebius Token Factory talks, sees and decides, and its answers move the robot's head. Nine models run on the computer beside the robot: NVIDIA Parakeet hears, Qwen3-TTS speaks, and NVIDIA Cosmos Reason 2 can run vision locally (it answered for a robot camera frame with the cloud eyes blocked, not yet in a spoken turn), with local models for the wake phrase, reading letters and finding a face or a person.
  • Nebius: Token Factory answers every turn: a median of 1.06 s from your words being understood to Andy's first sound, 54 turns on the robot. Serverless Jobs run Andy's search in Pollen's MuJoCo simulation, scored against the scene's truth, and the tests; a Serverless Endpoint, started when needed, serves NVIDIA Nemotron 3.5 Content Safety as a second check on what the person says.
  • Design: one household, three screens: the TV beside Andy, the family's phone page, a helper's settings page. Built for accessibility, to be read and heard across a room: large type in Atkinson Hyperlegible (the Braille Institute's typeface, every letter easy to tell apart), 17 px or larger on the phone pages, captions on by default at 7:1 contrast, a slower pace on request, louder at "I can't hear you", and talking over Andy stops it. Installs as a Reachy Mini app; try the screens, no keys.
  • Limits, said plainly: in 9 scored tries at home (2026-10-20) it pointed at hidden glasses 3 times, never at the wrong place; informally, it found real glasses in all five searches (three on 2026-10-28, 17 to 23 s; two filmed on 2026-10-29, about 21 s) and keys once, and the 30-try measure is still to do. No study with older adults yet. The video shows the real robot: each robot shot is one uncut take at 1× speed, the middle one 70 seconds.
  • Testing access: short testing instructions: guided examples with real Nemotron replies and the screens on sample data, both without keys; reproducible software checks; optional model evaluations; and a free live robot demonstration on request.

Inspiration

Three in four US adults aged 50 and over want to keep living in their own homes (AARP, December 2024), and their families want to help without taking over. Voice assistants answer questions, but they don't look at you, don't read the letter in your hand, and don't tell you which of your family looked at what you shared. We wanted a small robot that is useful every day, good company, honest about being a robot, and that keeps no recordings or transcripts: only what you ask it to keep.

What it does

Andy is a Pollen Robotics Reachy Mini that keeps looking at you while you talk. Three interactions bring the camera, moving body and TV together. The video shows all three on the real robot, each robot shot one uncut take; each interaction's test status is in the repository's live-test checklist:

  • "Can you read this for me?" You hold up a letter. Andy reads it and the TV shows it in large print. If the page asks for a fee or gift cards, or rushes you, code, not the model, puts Be careful first: "Don't call or pay yet. Check with family first." Then Andy offers to look it up, and the TV shows the consumer-protection warning Tavily found, with its source.
  • "Where did I leave my glasses?" Andy looks around the room, reads each view zoomed in, turns its head straight at the best sighting, checks a close-up, and only then points and says where. If it isn't sure, it says it couldn't see them. Nothing was trained: a vision model's answers move the robot.
  • "Could you look into whether the farmers market is on this Saturday?" Andy says it will, and carries on talking with you while the TV shows Tavily's research step by step; within a minute it tells you the answer and where it found it.

Andy also checks a message you only mention ("a man called saying my grandson is in jail"), keeps reminders in your words, every day if you like ("every morning at eight, take my pills": a reminder, never a dose), turns a family photo into a story, speaks English and Spanish, waves its antennas back when you wave, and remembers where you put your keys if you say yes. Family see only what you share, and Andy tells you who looked. Tried on the robot: the message check, reminders, waving back and the keys. Tested in code and the conversation suite, not yet tried on the robot: a daily reminder, the spoken half of a photo story, sharing a note by voice, turning to a second person speaking, and a move taught by hand.

How we built it

  • At home: NVIDIA Parakeet TDT 0.6B v3 and Qwen3-TTS on the home computer's GPU, so your voice never leaves the house, only words do; Silero VAD and Smart Turn v3.2, Vosk, Tesseract and YuNet on its CPU; and NVIDIA Cosmos Reason 2 for local vision; the conversation still calls Token Factory.
  • Thinking on Nebius Token Factory: NVIDIA Nemotron 3.5 Lightning talks and calls 34 of Andy's own tools, plus six of Pollen's (following a face, staying still, sleep, forgetting a fact, checking or cancelling a running task); NVIDIA Nemotron 3 Ultra takes hard questions and judges the tests. Zero data retention is on.
  • Seeing: NVIDIA Nemotron 3.5 Super VL on Token Factory reads each of four views as six zoomed-in parts, all 24 at once in about 1.8 s, then a strict yes or no about a close-up. On one room photo, zooming in took the search from 0 of 9 to 9 of 9 (a direction, not a rate).
  • Seeing offline: NVIDIA Cosmos Reason 2 8B on the home computer answers the same questions: in 30 simulated searches it pointed at the hidden object 22 of 23 times (Super VL 21 on the same trials), less precisely (1.95° against 0.78°), and never claimed a find that wasn't there; tried on the computer, not yet on the robot.
  • The body: Pollen's Reachy Mini conversation app is the agent runtime; Andy's profile, tools and body behaviours plug into it without forking it. What must always hold is enforced in code, not left to the model: the scam warning, consent before anything is kept or shared, and the 911 or 988 line when someone says urgent words.
  • Measured as Nebius Serverless Jobs: 97 spoken situations judged by Nemotron 3 Ultra, nine runs, every failure published; and Andy's own search in Pollen's MuJoCo simulation, a glasses-sized object hidden at random: with the close-up check, no false find in 7 searches without the object, against 3 in 7 without it.
  • A second check on the person's words, NVIDIA Nemotron 3.5 Content Safety on a Nebius Serverless Endpoint, caught 7 of 8 urgent phrasings our word list missed.

How Andy uses Tavily

Five of Tavily's APIs, each where it fits: search for local news and for scam warnings on consumer-protection sites; extract for links family send; crawl for what's on this week; map to find the town's events page at setup; and research, streamed step by step to the TV for "look into it for me".

What Tavily adds to the scam check, on 120 messages written as a person would read them to Andy, three runs as Nebius Serverless Jobs: Tavily's warnings, matched by Nemotron 3 Ultra, caught 58 of 60 scams and flagged none of 60 ordinary messages, naming a source in 170 of 174 verdicts. Each model alone caught slightly fewer and also flagged none, so what Tavily adds is the named source: the warning the person can check, from a site they trust. Matched by the smaller model, the same warnings flagged 24 ordinary messages, so only the stronger model keeps that source without false alarms. About $17 per 1,000 checks.

Challenges we ran into

  • A topic is not a trick. A grandchild's recital invitation and a "grandchild in trouble" scam share their words. One broad question flagged 11 of 40 ordinary messages; two narrow questions, the second asked of a stronger model, flagged none of 60.
  • Our judge passed a reply it should have failed: Andy read out a scam letter's phone number. Words that must never be said are now checked in code, and Andy never sees a letter's account or phone numbers.
  • Privacy in someone else's code, twice: a speech component printed what it heard to a console. We filtered its prints; an update then added a new one, and words reached our log for ten days before we found them. Now the health check reads the log itself and fails on any line that could hold what was heard.
  • A crisis line we nearly missed: at first "everyone would be better off without me" got no crisis line in 2 of 3 runs, and none in Spanish. A rule and a check in code fixed it, tested not to fire on grief.
  • A fast model drifts in small ways. Nemotron 3.5 Lightning answers in about a second, which a conversation needs, but it sometimes wrote a tool call out as text, skipped a memory offer the profile asked for three times, and said a tool's own reasoning aloud. Each is now caught in code: the written call is made, a one-line note on the turn asks for the offer, and tools give words to say. Over nine runs as Nebius Serverless Jobs the suite went from 93.2 to 94.8 of 97 on average, with batches of 94.2 and 95.1 in between, so the last steps are within run-to-run variation (decisions 0043 to 0045).

Accomplishments that we're proud of

  • A language model that moves a robot's head to find things, with nothing trained: it looks around, zooms in, turns to look closer, checks and points, and in its scored tries at home it never pointed at the wrong place.
  • A scam check with no false alarms and a source for nearly every warning: Tavily's consumer-protection warnings, matched by Nemotron 3 Ultra, flagged none of 60 ordinary messages and named its source in 170 of 174 scam verdicts.
  • Rules in code under the model: Be careful first on a risky letter, a letter's numbers never reaching the model, one yes for one exact save or share, and the 911 or 988 line on urgent words.
  • Every number has a dated scorecard in the repository, failures included, and Andy claims only what it shows.

What we learned

What older adults want from a home robot, across 21 studies (a 2026 systematic review): voice, control over their privacy and data, companionship and help with daily tasks. Andy is built around that, and leaves out fall detection and health monitoring on purpose. Measure the product, not just the model: Andy's real profile and tools, three runs each, beside a baseline. And put hard rules in code and judgment in the model. Two smaller lessons: a one-line note on the turn fixed what three paragraphs of the prompt could not, and a model told why will say the why, so tell it what to say.

Limitations: what doesn't work yet

  • Finding things at home misses more than it finds. In 9 tries on the robot, Andy pointed at the hidden glasses 3 times and never at the wrong place; all 6 misses were a pair on a chair beyond the head's reach, which the sweep saw in 5 of them but the closer look didn't confirm. Andy now turns its body to look past the head's reach; that is checked against Pollen's kinematics, not yet tried on the robot.
  • No study with older adults yet; a pilot with an ethics review comes next.
  • The current profile scored 94.8 of 97 on average over nine runs as Nebius Serverless Jobs (93 to 96; the nine runs earlier on 2026-10-28 averaged 93.2), with the real confirmation gate and synthetic tools. A photo story's memory offer fails most runs: Andy offers, but the judge wants a warmer word first. Nemotron still sometimes writes a tool call out as text; Andy makes the call when it fits one of its tools, and otherwise that turn says nothing. No unapproved write effect was reported. The 30-try physical finder measure is still to do.
  • It needs a computer with an NVIDIA GPU beside the robot, not yet a Jetson; English and Spanish only.
  • Our numbers are our own and small: the scam check's 120 messages were written by us for the test, not taken from real households; the suite is judged by a model; and the cost is an estimate from list prices, not a bill.
  • The local vision fallback needs a GPU of its own (the 8B model a whole 24 GB one) and hasn't been tried in a spoken turn on the robot with the primary vision service blocked. The voice conversation and tool selection still need Token Factory: this is not a complete offline assistant. GR00T and Sonic are for robots with arms or legs.

What's next for Andy

A pilot with older adults and their families; more languages (Parakeet already hears 25); and the Jetson version below.

Path to shipping

  • Hardware, once: a Reachy Mini Lite ($399 listed by Hugging Face, read 2026-10-25), a TV the household already has, and today a computer with an NVIDIA GPU (8 GB for hearing and speaking; more for the offline eyes), which most homes don't have. That computer is likely the biggest cost, and the reason for the Jetson plan.
  • Running, monthly: about $2.30 a month, estimated from list prices for one household's conversation and web lookups, not from a bill; the eyes add about half a cent a search. The second check is not in that figure: its endpoint is started when needed, about $1.55 an hour while it runs.
  • A Jetson port needs: hearing and speaking in well under 7.6 GB (a smaller voice than Qwen3-TTS), thinking and seeing staying on Token Factory, and Cosmos Reason 2 2B for when the network is down (about 4 GB of weights; what it needs on a Jetson is not yet measured). Planned, not tried on a Jetson.
  • Who pays: we expect the family who help, not the person; that is a guess the pilot has to test.

How to test it without a robot

Guided examples with real Nemotron replies, no keys: https://andydm-andy-companion-judge.hf.space/. The screens on sample data, no keys: https://huggingface.co/spaces/andydm/andy-companion. Andy's search in simulation: a 20-second film there and in the README, rerun with a Token Factory key. The real robot: live on a video call until judging ends (ask here), and the hardware on request.

Built With

  • cosmos-reason-2
  • hugging-face
  • mujoco
  • nebius-ai-cloud
  • nebius-serverless-endpoints
  • nebius-serverless-jobs
  • nebius-token-factory
  • nemotron-3-ultra
  • nemotron-3.5-lightning
  • nemotron-3.5-super-vl
  • nemotron-content-safety
  • nemotron-nano-vl
  • nvidia-cosmos
  • nvidia-nemotron
  • nvidia-parakeet
  • opencv
  • pollen-robotics
  • python
  • qwen3-tts
  • reachy-mini
  • tavily
  • tesseract
  • vllm
  • vosk
  • wikipedia
Share this project:

Updates

Submission history