Robot discoverer concept
Gazebo and AWS-robomaker-small-house-world (simplified) used as testing environment
High level diagram
Simulation services tab
Discovery tab to start and manage object detection and database
Direct control panel and chat to give instructions to the robot and query de knowledge database

Robot Discoverer

Inspiration

SLAM gives a robot a map of where the walls are — a grid of occupied/free cells with no idea what any of it means. We wanted a robot that could answer "where's the fridge?" or "take me to the kitchen," not just "here's a black-and-white occupancy grid." That means building a second, semantic layer on top of the geometric one: rooms with types, objects with descriptions, all grounded in the same coordinate frame the robot navigates in. And because a home-exploration robot's camera inevitably captures people, we wanted privacy to be a first-class constraint, not an afterthought bolted on later — every photo gets its faces blurred locally before it ever reaches a remote model or gets shown in a browser. Qwen Cloud's OpenAI-compatible endpoint made it possible to swap in a full vision-language model for the "what is this, and what room is this" questions that a fixed YOLO label set alone can't answer.

What it does

Robot side (Gazebo + ROS2 simulation, TurtleBot3 Waffle):

Builds a live occupancy-grid map via slam_toolbox, navigates it with Nav2/AMCL.
Runs an autonomous discovery tour: drives to a sequence of generated waypoints, rotates in place at each one, and photographs its surroundings.
Detects objects locally with YOLO11 on every frame (no API cost), fuses each detection's pixel bearing with the LiDAR range at that bearing to compute its real-world (x, y) position.
Blurs every face in every captured photo before anything else touches it — before the description model sees it, before the dedup embedding is computed, before it's saved to disk, before is sent to an external VLM/LLM. There is no un-anonymized copy anywhere, by design.
Calls a Qwen vision-language model exactly once per genuinely new object (never for a duplicate) to turn a bare YOLO class label into a real description — color, material, distinguishing features — and once per waypoint to classify the room type.
Segments the map into rooms (watershed segmentation) and re-segments automatically once objects are known, so furniture/clutter that a 2D LiDAR alone would misread as a room-dividing wall gets corrected retroactively.
Deduplicates objects re-seen from a different angle using both position proximity and a visual-similarity embedding, so re-running the tour merges into the existing world model instead of duplicating it.

Platform side (FastAPI web app, 3 tabs):

Services — start/stop the whole simulation stack (world, spawn, ROS↔Gazebo bridge, Nav2) from the browser; every process is PID-tracked and survives a webapp restart without double-starting.
Discovery — start/stop the tour, watch a live map with room polygons and waypoints, optionally require human confirmation before a "new" object is added (interactive mode) or let it run fully autonomous.
Control & chat — manual teleop with a live camera feed, and a LangGraph chat agent (also running on Qwen) that answers questions against the knowledge base ("how many rooms does the house have?", "where is the fridge?") and can actually drive the robot there ("take me to the kitchen" → a real Nav2 goal).

How we built it

Simulation: Gazebo (Harmonic) + ROS2 (Jazzy), a furnished flat world, TurtleBot3 Waffle with camera + LiDAR, slam_toolbox for mapping, Nav2/AMCL for localization and navigation.
Perception: YOLO11 (Ultralytics, CPU-only build — the GPU is busy with Gazebo's renderer) for detection; bearing/LiDAR fusion for 3D position; a local MobileNetV3-Small embedding for visual-similarity dedup; a Qwen vision-language model (via an OpenAI-compatible endpoint) for natural-language object descriptions and room-type classification.
Privacy pipeline: mediapipe's BlazeFace full-range face detector, chosen after benchmarking it against an OpenCV Haar cascade and BlazeFace's short-range variant on real test photos — full-range was the only one with usable recall on a wide, across-the-room shot. Each detected face is blurred as a Gaussian-feathered ellipse, tuned to a low confidence threshold because a missed face is a real privacy failure while an extra blurred lampshade is a harmless false-positive.
Knowledge layer: rooms as GeoJSON polygons, objects as GeoJSON points, both in the same map frame the robot navigates in, persisted in SQLite with GeoJSON export.
Chat agent: LangGraph react-agent with tools bound to the knowledge DB and to Nav2 (navigate_to_room), running on the same Qwen chat model.
Web app: FastAPI, server-sent events for live camera/pose/status/chat streaming, vanilla JS on the frontend (no build step, no framework), all backing processes registered as PID-tracked managed subprocesses.
Packaging: one Docker image/container for the whole stack, with a provider-agnostic model config layer — the same code runs unmodified against OpenRouter (Qwen models) or against Alibaba Cloud's DashScope/Qwen Cloud endpoint, just by changing MODEL_ENDPOINT/model names in one .env file.

Challenges we ran into

Sharing a single GPU between simulation rendering and perception. Gazebo's renderer already needs the GPU, so object detection had to run well on CPU alone without becoming the bottleneck of the discovery tour.
Telling furniture apart from walls using only a 2D LiDAR. A single-pass segmentation can't distinguish "a chair is here" from "a wall is here" from range data alone, so room segmentation needed a second pass, informed by the objects already discovered, to refine the first geometric guess.
Making the navigation stack safely controllable from a browser. Starting/stopping Nav2, SLAM, and the discovery tour as ordinary button clicks — with state that survives a web app restart — needed real process lifecycle management, not just running a script.
Deduplicating objects seen from unreliable angles, e.g. a wall-mounted object whose estimated position shifts between visits. Position proximity alone isn't enough, so dedup also uses a visual-similarity embedding as a second signal. Still a lot of work to do in this area.
Getting real face-detection recall on a wide, across-the-room shot rather than a close-up selfie photo — this took evaluating several detector options against real test photos and tuning the confidence threshold specifically for this camera distance and angle.
Supporting more than one LLM provider from day one, so the same agent/vision code runs unmodified against OpenRouter or Alibaba Cloud's Qwen Cloud endpoint — verified end-to-end against both.
Packaging a heavy, mixed ROS2 + ML Python stack (Gazebo, YOLO/PyTorch, FastAPI, LangGraph) into a single reasonably-sized Docker image, including choosing a CPU-only PyTorch build instead of the multi-GB CUDA default.

Accomplishments that we're proud of

A complete pipeline from raw geometry to natural language: SLAM → semantic room segmentation → autonomous object discovery → a chat agent that can query and act on the result.
Every object's position comes from real sensor fusion (YOLO bearing + LiDAR range), not just a 2D image-space box.
Privacy is structural, not a filter applied later: no un-anonymized photo is ever written to disk or sent to a remote model, and that was verified by design (the anonymizer runs before the crop is touched by anything else), not just tested after the fact.
A two-tier perception design — cheap local detection running constantly, an expensive VLM call only once per genuinely new object — that keeps API cost and latency down without losing descriptive quality.
Automatic, self-healing dedup: re-running the tour after the map grows or furniture moves merges into the existing world model instead of duplicating it, and a first-pass segmentation mistake corrects itself once objects are known.
A live, browser-based control room for the whole system — not just a demo script, a real 3-tab web app with PID-tracked services that survive a restart.

What we learned

Splitting local, fast, cheap perception (YOLO) from expensive, occasional semantic understanding (a VLM call once per new object) is a much more scalable pattern than calling a vision-language model on every frame.
Reasoning-capable LLMs need their reasoning budget accounted for explicitly in the token limit, since hidden chain-of-thought competes with the final answer for the same budget.
2D LiDAR alone is not enough to distinguish "wall" from "large object" — that ambiguity is best resolved with a second pass once more information (discovered objects) is available.
Privacy constraints are easier to get right when they're structural (blur before anything else touches the data) instead of a checklist applied after a feature already works.
A single shared container can be the simpler architecture over two coordinating ones, once both stacks need to talk to each other constantly.

What's next for Discovery

Move from the Gazebo simulation onto real hardware (a physical differential-drive base with camera + LiDAR).
Persistent, multi-session memory — recognize "this is the same house I mapped last week" instead of starting fresh every time.
A voice interface for the chat agent, so questions/commands don't require the keyboard.
Multi-robot support: several robots contributing to and querying the same shared knowledge base.
Richer temporal queries ("what's changed in the living room since last week?").
A pre-built, published Docker image so trying the project doesn't require building it locally first.