Kin

A small robot companion for parents who live alone: it checks in, nudges about reminders, reads letters for scams and keeps the family in the loop. Powered by NVIDIA Nemotron on Nebius.

Comment

Inspiration

A phone reminder is easy to ignore. A small robot that looks up at you and asks a question is not.

Kin is for two people: an older adult living alone, and the relative who worries about them from another city. About one in four Americans over 65 lives alone (ACL, 2023 Profile of Older Americans). The relative wants to know that Mum is up and about, that the pills were taken, that the letter promising two and a half million dollars went in the bin. Mum wants company, not a camera. Neither wants a medical device, and Kin is not one: reminders are a nudge and a log, and a quiet morning is "I haven't seen her yet", never "fall detection".

We had a DDL Vector 2.0 on the desk, a robot with a face, a camera and more personality than any smart speaker, and a cloud of open NVIDIA models a base URL away. The question was whether a palm-sized robot with a brain on Nebius could be a product rather than a demo.

What it does

Kin runs unattended, all day, on a mini PC beside Vector's charger. Five loops, four of them in the video, all of them live:

  • Daily check-in. At a set time he finds the person, greets them by name, asks how they are, and follows up on what he remembers ("I remember Vlad visited you yesterday, how did that go?"). Then the weather, with advice on what to wear for the walk.
  • Reminders that ask and listen. Pills, water, a walk, set by voice. When one is due he looks up, asks, and listens for the answer. No answer: he asks again ten minutes later, and after three tries the family gets a short Telegram message. Every outcome is logged, nothing is interpreted.
  • Presence. Through the morning he quietly looks for a face from his charger. No sign of the person by a deadline, and the family is told, and told again when she turns up.
  • Family bridge. A relative texts the Telegram bot; Kin delivers the message in person and sends the spoken reply back. Every evening the family gets a short note written from the day's log, and a weekly note on patterns.
  • Scam check. Older adults are the main target of mail, phone and text fraud, and new scams appear weekly. Hold up the letter or describe the call: Kin says "please don't do anything yet", searches the live web with Tavily for matching scam reports, and says plainly what he found. He may say a message matches a known scam or that he could not confirm it. He never says a message is safe.

And every night, a large model audits what Kin said that day against his own rules: medical advice, promises he cannot keep, implausible numbers, nagging.

How we built it

Thin robot, cloud brain. Vector senses and moves. A Linux mini PC (the hub) keeps the connection, the clock and the memory. Every decision is a call to a Nemotron model on Nebius Token Factory through the OpenAI-compatible API. Three Nebius products, each for the job it fits: the shared Token Factory API for the per-token live and nightly models, a dedicated Token Factory endpoint for the vision model that the shared API does not serve, and a Nebius AI Cloud serverless container for the nightly job, which is plain CPU work that should not rent a GPU.

  • Nemotron 3 Super is the live agent: a tool-calling loop with about twenty tools (say, ask, look_for_face, remember, set_reminder, weather, web_search, notify_family, go_home…). The model picks one action, sees the result, picks the next. Nobody programmed "if the face search fails, call out instead"; the model sees the failure and adapts. With server-side prompt caching a step takes 1 to 2 seconds.
  • Nemotron 3 Ultra does the slow thinking at 03:00: memory consolidation, weekly trends, the family's note and the self-check. That night runs in a Nebius Serverless container (CPU, cents an hour, five Python files uploaded as-is on a stock python:3.12-slim); the hub posts the day's log, gets back a list of edits and applies them to the local database. The whole night takes under a minute.
  • NVIDIA Cosmos 3 Super Reasoner on a dedicated Token Factory endpoint is his eyes: it read a tilted photo of a prize letter, with the amount, the fee, the deadline and the phone number, in 2.5 s.
  • Qwen3-Embedding-8B makes memory search semantic, with a keyword fallback so memory still works offline.
  • Tavily supplies fresh facts (scam reports, news); Open-Meteo and the National Weather Service supply numbers with explicit units. Numbers a person acts on never come from a search snippet.

Why live search is load-bearing, not decorative. A language model knows the scams in its training data. The letter on Jennie's table was printed last week, and the phone call she got this morning uses a script that did not exist a month ago. Older adults are the main target of mail, phone and text fraud, and the scripts change faster than any model is retrained, so a scam check that answers from memory is a guess dressed up as advice. That is the job Tavily does in Kin: when the person holds up a letter or describes a call, Nemotron calls web_search, Tavily returns clean, sourced passages about that scam, and Nemotron reads them and answers. The division is strict: Tavily supplies the fresh facts, Nemotron does the reasoning, and a safety rule in the prompt caps what he may conclude. He can say "this matches a known scam" or "I could not confirm this". He can never say "this is safe", because a false "safe" is the one error that costs someone their savings. The same tool answers the everyday questions a model cannot know (local news, TV listings, opening hours, the team's score), with the town and state forced into the query after "San Jose" once returned news from Costa Rica. Numbers a person acts on, such as temperature and times, never come from search snippets; those go through structured sources, after a snippet once gave Celsius labelled Fahrenheit.

Private by design. Speech recognition (Whisper) and the voice (Piper) run offline on the hub through a USB speakerphone, so no audio leaves the home. Camera frames are analysed and never stored. Memory is a SQLite file on the hub. The family note is written from one day's log, and Kin tells the person what he is sending before he sends it.

Built to be left alone. Every robot call has a deadline and can be cut short by a pet on his back. A dropped link is noticed within 30 seconds. Family messages are written to a durable inbox before Telegram is acknowledged, after one was lost in memory. Each alert is sent once a day. 236 tests run without a robot, a network or API keys. Kin ran unattended for weeks on the hub, including nights, and the git log is largely a record of what broke at 09:01 and how it was fixed.

Challenges we ran into

  • The robot is a 2018 toy. The SDK does not expose the microphone on production bots, the wake word triggered once in a day of trying, holding SDK control freezes his personality, and his camera sits 3 cm above the table. So Kin starts most conversations himself, listens through the hub's microphone after every question, borrows control per task and releases it, and asks people to hold things up.
  • Picking the planner. Nemotron 3.5 Lightning answers in 0.4 s against 1.7 s for Super, and on a first tool call both looked fine. On whole tasks on the robot, Lightning emitted a tool call as plain text, resolved "Friday" to today, and promised reminders it never set. Super with thinking off stopped calling tools at all. Lesson: judge a planner by complete multi-step tasks, live, not by a benchmark.
  • Vision on NVIDIA. No Nemotron on the shared API accepts images, and a dedicated Nemotron Nano V2 12B (vision) endpoint returned sixty colons at temperature 0, byte-identical across a redeploy. Cosmos 3 worked first time. Both findings are in our feedback notes.
  • Small things that end conversations badly. "Anything else?" answered by "That's it, darling, have a lovely day" fell through a list of ways to say no, and he drove off without a goodbye. The rule is now the opposite: after an offer of more, only a question or a request keeps the conversation going; everything else gets the goodbye. Whoever spoke last is owed a reply.
  • Physical reality. A window behind the charger turns its marker into a silhouette and he circles reporting "cannot see the charger". A PC USB port cannot power the dock. DHCP moved him mid-week and nothing could connect. The hub PC's Modern Standby froze everything after five idle minutes. Each one cost an evening and is now a line in the README.

What we learned

  • A robot that looks up at you changes the interaction. People answer him. The same reminder as a phone notification would be swiped away.
  • An agent loop over a handful of honest tools beats a scripted behaviour tree for a robot in a real home, because the home never matches the script.
  • The big model is most useful at night. Ultra auditing the day's transcript caught a temperature stated in the wrong unit and a four-day run of missed reminders, without once drifting into health interpretation.
  • Logs over interpretation. Kin reports the person's words and the outcomes; the family and the person draw the conclusions.

What's next

Tap-to-confirm on Vector's cube for people who would rather press a big button than talk. Better face finding (he still misses a face that is right in front of him). "Not yet" as its own reminder outcome, so he asks again later instead of logging a decline. A second person in the house. And a Jetson at the hub, so the voice, the ears and perhaps the planner can live at home too.

Kin is open source, MIT, and runs on any Vector paired with wire-pod: https://github.com/UBayouski/Kin

Built With

  • anki-vector-sdk
  • grpc
  • httpx
  • linux
  • nebius-ai-cloud
  • nebius-serverless
  • nebius-token-factory
  • nemotron-3-super
  • nemotron-3-ultra
  • numpy
  • nvidia-cosmos
  • nvidia-nemotron
  • open-meteo
  • openai-api
  • piper-tts
  • pytest
  • python
  • qwen3-embedding
  • sqlite
  • systemd
  • tavily
  • telegram-bot-api
  • vector-robot
  • whisper
  • wire-pod
Share this project:

Updates

Submission history