wherewolf.

3D Model

Inspiration

Everyone has torn a room apart looking for keys that were sitting in plain sight. On a computer, finding what you need takes a quick Ctrl+F, but physical living spaces have never had an equivalent. For most of us, misplacing an item is an annoying inconvenience that makes us late to class or work.

For older adults, people living with memory loss, and low-vision individuals, it is an everyday barrier to independence. Essential items like prescription glasses, hearing aids, and daily medication are easily misplaced, and existing smart tags only help if you remember to attach a battery-powered tracker to everything you own. Audio-only smart assistants can announce that an item was seen, but hearing an answer like "it's somewhere on the table" does little good if you cannot scan the room to locate it.

We built Wherewolf to bring Ctrl+F into the physical world. It's a companion that doesn't just tell you where your things are, but physically points them out.

What it does

Wherewolf is an ambient physical retrieval assistant that watches your living space, remembers where everyday objects are placed, and guides you to them with a laser pointer and natural speech.

For the user: ask out loud, "Where are my keys?" The robotic wolf swings its head across the room, locks an onboard laser pointer onto the target, illuminates a 16-LED ring on its belly in the direction of the item, and speaks an answer grounded in room landmarks: "Your keys are on the desk, left of the laptop." You can speak hands-free, push a physical talk key, or search directly from a live web dashboard.

Visual memory and teach-by-showing: Wherewolf monitors the room continuously. It tracks everyday essentials, such as keys, earbuds, phones, wallets, chargers, recording their spatial coordinates and cropped visual snapshots. When introduced to an unfamiliar item like a retainer case, the user simply holds it up and names it; Wherewolf registers its visual signature on the spot without retraining.

When an item is hidden: if an object is moved out of view or placed inside a backpack, Wherewolf recognizes that it is gone, announces exactly when and where it was last seen, and displays the historical snapshot on the dashboard.

Video never leaves the local machine. Object detection and spatial indexing run entirely on-device, only cropped thumbnails of detected objects are stored, and an instant-wipe button purges the database on command.

How we built it

The host computer runs the real-time computer vision, spatial memory, and speech, while an ESP32-S3 handles kinematic pan/tilt actuation, closed-loop laser steering, and LED feedback over a dedicated wireless link.

The Stack:

Vision and Detection: A TP-Link Tapo 1080p camera streams room video over RTSP. Open-vocabulary detection is powered by Hugging Face's OWLv2, running in FP16 precision directly on the Mac's GPU at approximately 0.23 s per pass. To catch small, distant objects, an alternating spatial crop zooms into one quadrant of the frame per pass. A custom centroid tracker and Lucas-Kanade optical flow maintain stable object tracks between inference passes.
Embedded and Kinematics: An ESP32-S3 controls two SG90 servos mounted on a 2-axis pan/tilt bracket, a 650 nm red laser diode, and a 16-LED WS2812 addressable ring. The microcontroller communicates with the Mac over a local Wi-Fi link via authenticated UDP packets. Servo trajectories glide at 100°/s with single-direction approach logic to eliminate mechanical gear backlash.
Voice Pipeline: Spoken audio routes to Gemini 3.5 Flash-Lite, which parses raw speech into structured JSON action schemas (find, teach, forget, rename). Because the model only extracts intent, it cannot hallucinate an object's location. A local Faster-Whisper engine runs in parallel as a zero-latency fallback. Audio responses are synthesized via Gemini TTS streaming or macOS native speech offline.
Database and Spatial Indexing: SQLite stores canonical object labels, 2D normalized coordinates, observation timestamps, and cropped snapshots. A landmark-relative rule engine calculates spatial descriptions by evaluating bounding-box centroids relative to dominant furniture anchors.
Chassis and Hardware Enclosure: Custom case designed in OpenSCAD and 3D-printed in PLA. The case is styled as a wolf's head and covers all of the important hardware such as the ESP32, breadboard, and antenna
Web Dashboard: A local website serves a dashboard with live RTSP video feeds, Server-Sent Events (SSE) for real-time state synchronization, and an interactive Three.js and GSAP 3D intro.

Challenges we ran into

Small objects, far away. Standard YOLO-World detectors scored an unusable 0.003 confidence score on loose keys and earbud cases across a room. Switching to OWLv2 FP16 restored zero-shot recall. Introducing a 4-quadrant cyclical zoom improved detection confidence on distant keys from 0.34 to 0.45 and increased phone detections from 26 to 62 frames across identical footage.

Semantic look-alikes. A smartphone lying flat and viewed edge-on scored higher as a "remote" (0.56) than as a "phone" (0.30). Pruning "remote" from the active vocabulary allowed the model to correctly identify the phone in 18 out of 20 test frames.

Fisheye compression smoothing tiny laser dots. At a distance of 3 meters on a 1080p wide-angle camera, a laser dot spans only 1 to 2 pixels. Standard color thresholding failed because sensor saturation turned the dot's center pure white, and H.264 compression washed out static color cues.

API tail latency. While cloud voice APIs typically responded in 0.9–1.5 s, approximately 1 in 8 requests experienced network spikes of up to 15 s. We implemented an aggressive race condition: if the cloud voice call does not return an intent within 1.6 s, the system triggers local Faster-Whisper at 4.5 s, ensuring the user is never left waiting in silence.

Power bank low-current shutoff. Smart power banks automatically shut down when the idle servos drew too little baseline current. We added a dummy load cycle and shifted the WS2812 logic level directly to 3.3V to ensure consistent power distribution without brownouts.

Accomplishments that we're proud of

Zero-shot open-vocabulary indexing: users can introduce custom, unmodeled objects via teach-by-showing in seconds without dataset collection or model fine-tuning.
Fast, hybrid conversational interface: natural speech translates to physical actuation in approximately one second, with full offline speech fallbacks when internet access drops.
A fully integrated physical artifact: a self-contained, 3D-printed wolf housing compute, wireless power, and actuation in a sleek design.
Local-first privacy architecture: video feeds and spatial databases never leave local memory, retaining isolated object snapshots rather than continuous room recordings.

What we learned

Measure on your own sensors, don't guess. Models that look comparable on standardized benchmark papers diverged 100-fold on our specific wide-angle camera feed. Testing on recorded room footage settled model selection in minutes.
Closed-loop feedback beats open-loop mechanical precision. Budget micro-servos suffer from severe gear slop and lenses warp straight lines. It's better to calibrate your hardware instead of adjusting on the fly.
Build the software harness before the hardware arrives. Writing a simulated kinematic rig with artificial lens distortion allowed us to write, debug, and stress-test the visual servoing controller before physical parts arrived.
Tail latency defines user trust. An average response time means nothing if one out of eight queries hangs. Racing parallel cloud and local speech engines masked network unreliability completely.

What's next for Wherewolf

Rewind indexing: record short 3-second motion buffers whenever a tracked item is moved to answer questions like "Who moved my charger?"
Natural descriptive search: support fuzzy visual queries like "Find my blue water bottle" or "Where is the thing I plug my laptop into?" using multimodal vision embeddings.
Find by sound: integrate a 4-channel microphone array to locate buzzing phones or beeping smoke detectors and point directly toward the sound source.
Multi-room Coverage: sweep the camera across the pan/tilt head to construct a panorama memory, and let several devices share one memory across rooms.
Low-vision clinical partnerships: collaborate directly with occupational therapists and memory-care professionals to evaluate usability, voice pacing, and hands-free tactile routines.