Peeki is a Wi-Fi camera bear with a visual memory. An ESP32-CAM in a fluffy body describes what it sees, keeps those descriptions as memories in a database, and talks to you about them over text and voice calls through Relay. It's a cute Ring camera you can actually talk to, and one that only pays attention to what you told it to care about.

Inspiration

My Ring camera sends "motion detected" at 2 AM when it's the neighbor's cat. It can't answer the one question I actually have: "did anything happen there yesterday?" I wanted a camera that remembers what it sees in words and that I could ask like a roommate, plus a way to teach it what's worth noticing with one sentence: "tell me when someone walks in." The hardware started as my spare ESP32-CAM I never got around to using, so I gave it life with the Peeki mascot and brain.

How I built it

The loop is see → describe → remember → converse → notify.

The ESP32-CAM uploads VGA JPEGs to a FastAPI server every 250 ms over an authenticated WebSocket, tunneled through Cloudflare so peeki.tech works with no open router ports. Before uploading, the firmware compares 1/8-scale pixels against the last frame and skips upload when nothing moved:

$$\bar{d}=\frac{1}{N}\sum_{i=1}^{N}\left|p_i-p_i'\right|,\quad \text{upload only if } \bar{d}>0.035$$

The server adds its own gates (at most one description per 5 s, a 60 s refresh so a quiet room stays in memory), so Grok vision runs on meaningful frames instead of every one. Each description is stored as text in Neon Postgres; no permanent images or clips, which keeps memory small and easy to erase ("forget today").

One tool-using brain serves everything. The model gets ten tools: search observations, look through the camera right now, create/pause watches, save or forget memories, pause the camera, flash the torch. The same brain and conversation history answer Relay texts, voice calls, and the Svelte portal's debug chat. On Relay, a single WebSocket consumer handles events; event IDs are committed to Postgres before the ACK (Relay redelivers otherwise), and every send carries a idempotency key so a reconnect replay can't double-send. Watch matches become durable jobs in Postgres that the agent claims and composes into a short message in Peeki's voice, like "👀 someone just walked in."

Voice calls were the fun part. Peeki answers within Relay's 32-second join window, transcribes your voice with ElevenLabs Scribe, replies through the same tool loop on a non-reasoning Grok model to keep latency low, and streams ElevenLabs speech. Replies also render as captions drawn onto a 720×1280 video track with the live camera as an inset, plus 24 fps talking/listening sprite animations so the bear visibly reacts. Hang up mid-sentence and the queued audio cancels cleanly.

Challenges I faced

The big one was agent design, and it changed the whole architecture. My first version hardcoded the happy paths: an intent router mapping phrases to internal task types (IDENTIFY, MONITOR, REVIEW...), with hand-written branches for each phrasing (keys left on counter, "when was the last time", "stop watching the door"). It worked great in demos and fell apart the moment someone said it slightly differently or asked a follow-up that needed an ID from the previous turn. Every sentence became a new if-branch.

So I inverted it. Instead of teaching Peeki to parse, I exposed the same ten tools to the model through Relay and let Grok decide what to call, in a loop with up to eight tool round-trips. The tools carry the rules in their schemas and descriptions (a watch_id must come from list_watches first; save_observation takes an actual observation ID returned by search; a failed job gets reported honestly instead of claimed). The hardcoded router shrank to almost nothing and reliability went up, because the model handles phrasing variety natively while the tools guarantee actions only happen if their preconditions were actually met.