Team
Apparatus L
Apparatus F
Apparatus R
Laser Sweep Grid
Sample Object Permanence Detection

Ask the Room

Inspiration

What if your room could remember where you left things, even when you could not?

Anshul's grandfather and Guru's grandmother both lived with dementia. One thing that stayed with us was how often something really small could become frustrating.

"Have you seen my pills?"

Then a few minutes later:

"Where are my pills?"

We started wondering what it would look like if a room could remember for you.

At first, we thought a camera system would be enough. Then we realized that recognizing objects was only part of the problem. A camera can tell you what it sees right now, but it does not remember what happened after something disappears from view.

Our demo is based on that exact situation. Put the keys on the table, cover them with a notebook, put the notebook inside a box, and move the box across the table. A normal vision system now sees a box. We wanted ours to still know that the keys are in there.

That led us to build Ask the Room around object permanence. Instead of treating every video frame as a separate image, Ask the Room maintains a persistent model of the objects around it, how they relate to each other, and how those relationships change over time.

We wanted to build something that could give people a little more independence without asking them to open an app or search through a camera feed every time they lose something.

What it does

Ask the Room is a voice-controlled, on-device AI assistant that keeps track of everyday objects as they move through a space. Our world model currently supports eight objects: keys, a pill bottle, a wallet, glasses, a phone, a remote, a box, and a notebook.

For each object, the system maintains a state such as VISIBLE, HELD, UNDER, INSIDE, GONE, or UNKNOWN, along with relationships between objects. For example, it can represent keys → UNDER notebook → INSIDE box → ON table. If the notebook goes into the box, the keys follow it in the model. If the box moves, the keys move with it.

You can ask questions like "Where are my keys?", "What happened to my keys?", "Did anyone touch my pills?", or "What did I miss?" Location and history questions come directly from the world model and event log. When our rules cannot understand a question, we use Grok as a reasoning layer. It can interpret more natural or ambiguous questions, reason over the current world state, use the current camera frame for visual questions, and name unknown objects live. For example, Grok successfully identified an unknown object as an "earbud case" with 0.90 confidence.

We also wanted the system to do something physical with its answer. A red laser on a pan-tilt stepper turret points to the location of the object. Before a run, we sweep the laser across the workspace to build a map of where the dot lands, then use that map to correct each aim. During operation, the camera watches the actual dot and closes the loop on the final position. The system checks for people and hands before each aim, turns the laser off after at most four seconds, and uses a drift lock to shut it off if the aim shifts unexpectedly.

We have also built a first version of room memory beyond the tabletop. Whole-room questions have been tested on real frames from the rig, the phone map shows live sightings, and room pointing is working in a limited form. The couch handoff is still being tested on the physical rig.

How we built it

The core of Ask the Room runs on an NVIDIA Jetson Orin Nano 8 GB, making edge AI a central part of the system. Our camera is mounted in the corner of the room and captures 2560×1440 video, with the tabletop cropped to 1280×720 for the main table-processing pipeline. Object detection happens locally on the Jetson rather than sending the entire video stream to a cloud service.

Our main detector is a fine-tuned YOLO26 model trained on our own Brio photos and exported through TensorRT. On held-out overhead-view photos, it reached 0.995 mAP50 on the trained objects and 0.81 on hands. YOLOE-26 runs alongside it to propose boxes for objects outside the fixed training set. For the current camera, wallet, phone, notebook, and pill bottle are the main fine-tuned classes, while objects such as the keys in our demo can be treated as unknown objects and handled by the open-vocabulary path. The current YOLO26 model was trained from an overhead viewpoint, while our final camera is mounted in the corner, so improving that viewpoint is one of our next steps.

YOLO tells us what is visible, but the world model is what gives those detections meaning over time. We built a deterministic state model that handles hand contact, covers, containers, nested containment, disappearing objects, confidence, and decay. A single AprilTag provides the physical reference frame, and a homography maps the camera view into table coordinates that can also be used by the laser.

For voice, Silero VAD and whisper.cpp handle speech locally. A separate addressing layer decides whether nearby speech was actually meant for Ask the Room, and a rule parser handles common requests such as WHERE and HISTORY. Grok sits above this system and handles the cases where rules are not enough. It can interpret difficult language, answer open-ended questions using the structured state of the room, reason over visual input, and name unknown objects.

The physical system is built around an Arduino Uno driving two closed-loop stepper motors on the pan-tilt laser mount. The workspace sweep builds the laser map, and the camera provides visual feedback for each aim. This lets the hardware correct itself based on where the dot actually lands instead of relying entirely on a fixed calibration.

FastAPI powers a local dashboard with live video, the belief graph, and the event timeline. SQLite stores events and question history, while n8n handles logging and health checks and Twilio provides another way to send questions. The core perception, world model, speech recognition, and offline voice path can run locally, while Grok and ElevenLabs add capabilities when a connection is available.

Challenges we ran into

The hardest part was not getting YOLO to detect objects. It was figuring out what a missing detection actually meant. If the keys disappear, they could have been covered, picked up, put inside something, carried away, or simply missed by the detector. Those situations can look very similar from the camera, so we had to build explicit state transitions and tune them using replayed scenarios.

Nested containment made this even harder. Tracking keys → notebook is straightforward, but keeping keys → notebook → box correct while the parent objects move requires the world model to propagate the entire relationship chain. This ended up being one of the most important parts of making the object permanence idea actually work.

We also spent a lot of time figuring out how to use Grok without making it responsible for the physical state of the room. We wanted its multimodal reasoning capabilities, but we did not want an LLM guessing where an object probably was. Keeping the world model deterministic and giving Grok structured information to reason over gave us a much better balance between reliability and flexibility. It also gave us a way to handle objects outside our fixed training set.

The laser took some iteration as well. Our first calibration worked until the hardware shifted slightly, so we built a workspace sweep to map where the laser actually lands and then added closed-loop correction so the camera could see the dot and adjust the steppers. We also added safety checks so the laser shuts off when a person or hand is in the way, after four seconds, or when the system detects unexpected drift.

Getting all of this to run on one Jetson while detection, speech, and local processing competed for the same compute was another major part of the challenge. We had to balance model size, TensorRT acceleration, and responsiveness while keeping the world model fast enough to run continuously.

Accomplishments that we're proud of

We are most proud that Ask the Room became a complete physical system rather than just a computer vision demo. The camera sees the environment, YOLO detects objects, the world model remembers them after they disappear, the voice system understands the question, Grok helps when the language or visual reasoning gets harder, and the laser physically points to the result.

We are also proud of the edge AI side of the project. YOLO26 and YOLOE-26 run through TensorRT on the Jetson, while speech recognition, the world model, event history, and offline voice output all have local paths. The system can keep its main perception loop on the device instead of requiring the room to be continuously streamed to the cloud.

We saw 0 identity mix-ups across all recorded real clips. Across the project, we have run 3,007 automated tests.

The combination of the world model and Grok is another part we are especially proud of. The deterministic layer keeps track of what is physically happening, while Grok gives the system a natural way to understand questions, reason over the current state, use visual context, and handle objects that fall outside the fixed detector classes.

We are also proud of the first version of room memory beyond the tabletop. The same system can now connect live sightings and whole-room questions to a phone map, giving us a path toward a much larger spatial memory system.

What we learned

Our biggest lesson was that object permanence is really a world-model problem, not just an object-detection problem. Recognizing the keys is easy. The interesting part is understanding that the keys are still there after they disappear and that they should move with the notebook and then the box.

We also learned that LLMs become much more useful when they have a structured world to reason over. Instead of asking Grok to reconstruct everything from raw video, we give it the state that our system already knows and let it focus on language and reasoning. That made the system easier to control and easier to debug.

Running the system on the edge changed the project too. Using the Jetson meant the camera and perception loop could stay local, the world model could update continuously, and the system could react directly to physical hardware without sending every frame to a remote service. It also forced us to think carefully about model size, TensorRT optimization, and how different parts of the pipeline share compute.

The laser taught us a similar lesson on the hardware side. Calibration gets you close, but feedback is what makes a physical system reliable. Mapping the workspace with a laser sweep and then correcting against the actual dot made the system much more robust than simply trusting a fixed calibration.

We also learned to be careful about the difference between what the system observes and what it assumes. For example, Ask the Room can report that the pill bottle moved, but it does not claim that someone took the medication. That distinction became important throughout the project.

What's next for Ask the Room

Right now, Ask the Room combines the tabletop object-permanence system with a first version of whole-room memory, visual questions, room pointing, and live sightings on a phone map. We want to make that spatial memory reliable across an entire home so relationships between objects and rooms can persist as someone moves through the space.

We also want to retrain YOLO26 for the actual corner-mounted camera instead of the overhead viewpoint used for our current training data. That should improve real-world perception while keeping the model efficient enough for the Jetson. YOLOE-26 and Grok can continue helping the system understand objects outside the fixed training set.

We want the room memory to become continuous as well. A real home is always changing, so the system should keep updating what it knows as people move objects throughout the day instead of operating mainly in a controlled scene.

The laser can become more expressive too. Today it can point to a location using its workspace map and closed-loop correction. Eventually it could trace a path, guide someone to an object, or move between several locations as it explains an answer.

Finding keys is the first use case because it is simple and immediately useful. The bigger idea is a system that remembers the physical world well enough to answer questions like "Where did I leave my glasses?", "What changed while I was gone?", or "What did I move earlier?"