deixis: tell robots where

Robots understand what you ask, not where. Point or stand and say "here", "that one", "this big": one open core gives the robot the place. On a vacuum, a plotter and an arm, with Nemotron on Nebius.

Comment

Inspiration

There are crumbs under the kitchen table and you want the vacuum to clean there. Today: open the app, find the map, find the table, drag a box. With a person you would point and say "here".

Robot vacuums are already in homes: 23.5 % of 1,000 US adults surveyed own one [Reviews.org, Apr 2026]. Yet none of them can be sent there; one maker shipped "point and say clean this" in August 2026 [Matic Cues, 9to5Mac], for its own brand only.

What's new. "Here", "that one" and "this big" take their meaning from where you are and what your hand is doing; linguists call them deixis. MIT's "Put That There" did this in 1980 with fixed hardware in one room; deixis does it with any robot, on open code. Today's pointing models (Molmo, RoboPoint) hand back a pixel in a picture; deixis hands back a place in the robot's own millimetres, with how sure it is, over a tool call any agent can make.

What it does

The core, in one line: plain code that turns your words plus your pointing into a place in the robot's own millimetres, or into a question. No AI model in it, MIT, and any robot can call it over MCP (how AI agents call tools).

What this is: an open toolkit for robot builders, not a consumer product: one core, three robots, three kinds of "where". The pen plotter (a 3D printer rebuilt to draw) is our ruler: ink shows where it went. The vacuum is shop-bought and unmodified; the browser arm needs no hardware. It is for anyone who would rather point than open an app (a parent, a guest, someone with full hands) and for builders who put a language model in a robot and then build "here" from scratch. As far as we found, no open piece does this; the ones we would plug into are already on LeRobot, where our SO-101 comes from (about 28k stars, not requests).

Every robot, the same four things:

  • Guess: it shows and says where it thinks you meant.
  • Yes: it acts only on "yes".
  • Stop: "Stop" stops it.
  • How sure: a soft blue halo, tight for a steady hand, wide for a shaky one. When two places are about equally likely, it asks: "The red one or the green one?"

Pen plotter: a place and a size. Draw where you point, as big as you show.

  • What you see: point and say "Draw a kite here"; open two fingers: "this big". The pen traces that width as a circle, pen-up, asks "A kite, about four centimetres?", and draws on "yes". Picture 3.
  • Result (6 Oct): over 4 printed crosses, all one person's pointing, the ink landed a median 2.2 mm from where we pointed, worst 4.3 mm.
  • Caveat: four crosses is a thin sample. The cross fell inside the halo every time, which shows the halo is not too small, not that its width is calibrated.

Robot vacuum: an area. Clean without the app, on a shop-bought robot we didn't modify. The assistants it does take (Alexa, Google Home, Siri) start, stop and dock it; from what they publish, none says where.

  • What you see: stand at one corner, "Clean from here …", walk to the other, "… to here". The window draws the box on a camera view of your floor, asks "Clean from there to here?", waits until you step out, then cleans.
  • Result (8 Oct, 18 boxes): 77 % of the positions the robot reported fell inside the box it was given. A robot covering a box drives over the edge as it turns, so that is a floor on containment, not 23 % wrong.
  • Caveat: we never checked the camera's box against the corners we actually walked, and the 77 % is from the robot's own map, so its drift and our floor mapping are mixed in. Picture 8.

Robot arm: an object and a place. Pick and place, the everyday task of home and hobby arms, on an SO-101 simulated in your browser.

  • What you see: move your fingertip onto a block, "pop that one …", then a spot, "… over here", and it moves it.
  • Result (8 Oct, 30 commands in 9 different sentences): 28 of 30 ended correct; 23 of them needed no question and no correction (77 %).
  • Caveat: of the seven not right first time, two were mis-heard (the microphone caught nothing) and five took a question or a correction. The same few sentences repeat, so this measures the pointing more than the wording. No human baseline. Picture 9.

How we built it

  1. Hear each word and when it was said: NVIDIA Parakeet gives every word a start time.
  2. See the pointer: MediaPipe finds your fingertip or your feet, and when it held still.
  3. Match them by time: the hand usually gets there first, so deixis looks back up to 1.5 s before "here". The longest lead we logged, 1.48 s, set that window. The hand stopped before the pointing word in 15 of 17 commands, median 0.39 s earlier; in the other two it did not, and looking back cannot catch those.

In the live loop, and not. While you talk, two models run: Nemotron turns the sentence into tool calls, Parakeet gives the words their times. Cosmos is built into the plotter but is in no scored run (it hears no sound, so it could never have done the timing), and SANA drew its pictures in advance.

NVIDIA Nemotron 3.5 Lightning, on Nebius Token Factory, turns the sentence into tool calls, with reasoning off and all of a sentence's calls in one reply. That is why Lightning over Nemotron 3 Nano: Nano returns one call a reply, so "a house here and a tree there" loses the tree. And on our 61-command set Lightning got 52 right to Nano's 43, same price. It never writes a coordinate; it points at words:

place_drawing({
  "object": "kite",
  "place": {"deixis": "here", "word_index": 3},
  "size": {"word_index": 4}
})

The core turns "word 3" into millimetres, and has no robot in it: a test fails if "vacuum" or "pen" appears in its code.

Model or service Its job Where it runs
Nemotron 3.5 Lightning The request, as tool calls Nebius Token Factory
Parakeet TDT 0.6B v2 Words with start times The robot's computer (CPU); the web demo server
Cosmos 3 Nano Reads what is drawn on the paper, so "next to the house" has something to match. 89 % right on computer-drawn test sheets; never scored on the plotter's own paper Nebius AI Cloud Serverless Endpoint
SANA 1600M (NVIDIA Research) The plotter's drawings, 852 made in advance; a person picked the ones we use Nebius AI Cloud Serverless Job

Why Nebius. Every command goes to Token Factory first: one key, the OpenAI SDK, and about a second of the plotter's 2.1 s is the model call itself. Per-token suits a robot (one short call a command, no machine to keep warm), though we have not measured p99 or load.

  • Token Factory (Nemotron), at a list $0.06 per million prompt tokens: the arm's prompt comes to $0.34 per 1,000 model calls, the plotter's bigger one about $0.50. All our model-choice work (seven comparison runs plus smoke runs and probes) cost $0.22 of tokens.
  • Serverless Endpoint (Cosmos): Token Factory has no NVIDIA vision model (checked 9 Oct), so the step that needs eyes runs there, naming the sketches on the paper: built, not scored. The public endpoint suits a demo; a fleet would want a dedicated one.
  • Serverless Jobs (SANA): 852 drawings in about 24 minutes for about $0.72, on one RTX PRO 6000 at $1.80/h, run once as a batch so nothing sat idle.
  • A cost lesson (1 Oct). We left a Cosmos endpoint standing while we read its results and decided what to try next; it served nobody and billed by the second. Cold start plus that idle came to 67 % of the billed day, about 15x the busy-time cost per call. Now we delete the endpoint between sittings; scale to zero, or billing from the first request, would have made the mistake impossible.

Add your robot

Adding a robot took about 1,000 lines: the vacuum's whole side of it (cloud API, MCP server, window, profile) came to 1,033, written by one person who also wrote the core, so read it as an upper bound. Everything else is reused. Three steps:

  1. a profile: words, sizes, work area;
  2. an adapter: show the guess, act, stop;
  3. a pointer: a fingertip, your feet, a mouse click.

Safety: the model never moves anything; a hand on the paper pauses the pen; drawings that run off the page are cut, never moved; the vacuum waits until you are clear.

Challenges we ran into

  • Word timing. We tried NVIDIA's two streaming ASR models first: their last word landed up to 0.76 s late, and "here" is often last. Parakeet, with a fixed 0.1 s correction, is off by 0.05 s (median) on synthetic voices.
  • Real speech, not clean speech. Parakeet heard "to here" as "Two here" and "yellow" as "error"; the apps repair the transcript first.
  • Small sizes. Trackers put the fingertip mid-finger, so a small "this big" reads about 7 mm too big.
  • The vacuum's hidden zone command. The firmware refused the open library's area clean; we sent the official app's own message.

Accomplishments that we're proud of

  • From your last word to the robot's guess: a median 2.1 s on the plotter, 2.9 s on the arm and on the vacuum (6 and 8 Oct; 31, 29 and 31 commands).
  • Three robots on one core, with every number here measured on a dated run and a caveat where we could not measure.

What we learned

  • Geometry for precision, the language model for meaning. Pointing at words, not coordinates, is safer and testable.
  • Show the guess first. A wrong guess is cheap to fix.
  • Ask, don't guess.

What's next for deixis

  • Home robots with arms: even an arm on a vacuum takes "where to put the socks" from an app map; "Put that there" is the missing command (our reading, not a product we found).
  • People who can't use their hands: of 11 Kinova Jaco wheelchair-arm owners studied, 9 struggled with switching modes (PMC12353078). We have not tested this with any such user. Next: a gaze or head pointer.
  • More languages, and pointing from across the room.

Honest limits.

  • One person, tested by the maker: a case study, not a user study.
  • A coached protocol: point, stop, then speak, so the timing logs size the window rather than describe a stranger's pointing (Levelt 1985 finds the gesture leads in natural speech too). Holding still helps; by how much is not measured.
  • Flat surfaces only, English only, and the vacuum needs a camera that sees the floor.

Built With

  • deebot-client
  • fastapi
  • marlin
  • mediapipe
  • model-context-protocol
  • mujoco
  • nebius-ai-cloud
  • nebius-serverless-endpoints
  • nebius-serverless-jobs
  • nebius-token-factory
  • nemotron-3.5-lightning
  • numpy
  • nvidia-cosmos
  • nvidia-nemotron
  • nvidia-parakeet
  • nvidia-sana
  • onnx-runtime
  • opencv
  • pydantic
  • python
  • sherpa-onnx
  • three.js
  • typescript
  • vite
Share this project:

Updates

Submission history