Inspiration

Smart Rings started with a simple observation: a colleague frequently checked CCTV footage to keep track of her child. That made us wonder—could people describe what they care about and receive a relevant alert instead of repeatedly watching a camera feed?

We explored several smart-home ideas before focusing on natural-language monitoring. Porch and package monitoring gave us a practical starting point: detecting deliveries, noticing package removals, and checking whether a person is carrying a box.

What it does

Smart Rings turns plain-language monitoring requests into structured automation rules. Users can type a command or record a voice instruction in the browser, review the transcription, and create an alert.

Example requests include:

“Alert me if a person is carrying a box.”
“Alert me if someone wearing black pants is seen.”
“Alert me whenever a package disappears.”

The system evaluates supported conditions against video footage and sends matching alerts to connected Fire TV clients. It also exposes MCP tools so compatible assistants can manage rules and access the prototype’s history and deadline functions.

The current monitoring demonstration uses recorded footage. Live Ring-camera inference is a future integration.

How we built it

We built the backend with Python and FastAPI, the command dashboard with Streamlit, and the Fire TV application with React Native and Amazon Vega tooling.

An LLM accessed through OpenRouter converts commands into JSON rules. Pydantic validates the generated configuration before the automation engine accepts it. The rules are generated from the user’s instruction within the application’s supported schema.

Our visual pipeline combines:

YOLO for object detection and bounding boxes.
BoT-SORT for temporary person and vehicle tracking IDs.
Qwen2.5-VL through Ollama for targeted visual questions about object crops.
Roboflow for verification of candidate package-count changes.

We integrated faster-whisper for local speech transcription and the official MCP Python SDK for Streamable HTTP connectivity. Alerts reach the Fire TV application through the backend’s webhook and WebSocket flow.

Challenges we ran into

One challenge was making generated rules executable. Valid-looking JSON is not enough: every rule must match the evaluator’s capabilities. We added structured output, validation, and a repair attempt for empty or malformed model responses.

Another challenge was repeated detections. YOLO may detect the same person across many frames, so simply alerting on every detection creates duplicates. Suppressing the rule after its first match can instead miss another person. We introduced tracking and separate state for each track and visual rule, with periodic rechecks for changing conditions.

Package counting also required verification. A count can change because of occlusion or detection errors, rather than an actual delivery or removal. Our hybrid workflow checks candidate changes before accepting them as observations.

Finally, detailed visual reasoning adds processing time. We use selected object crops and scheduled checks rather than analyzing every frame with the vision-language model.

Accomplishments that we're proud of

We connected voice input, natural-language rule generation, visual monitoring, and a Fire TV alert interface in one prototype.

We moved beyond basic object detection by combining tracking with targeted visual questions. This lets the system evaluate conditions such as carrying a box while continuing to check how a tracked person’s condition changes.

We also exposed six MCP tools for rule management, event-history queries, and deadline alerts, making the project’s functionality reusable outside its dashboard.

What we learned

We learned that a detection is not automatically an event. Reliable monitoring requires tracking objects, verifying changes, evaluating conditions, and deciding when another alert is appropriate.

We also learned to give each model a focused responsibility. The language model generates supported configurations, YOLO locates objects, and the vision-language model answers specific visual questions. Application logic and validation connect those stages.

Just as importantly, users need clear feedback about whether a rule was created, monitoring is active, and an alert destination is connected.

What's next for smart rings

Our next priority is persistent storage. Rules, event history, and deadlines currently live in memory and are lost when the hosting process stops.

We plan to explore a time-series database for timestamped detections, verified count changes, and alerts. This would support richer questions such as “How many packages arrived yesterday?” or “Was someone carrying a box between 2 PM and 4 PM?”

We also want to connect live camera feeds to the inference pipeline, improve tracking continuity, build durable deadline scheduling, and strengthen authentication and alert delivery.

As we prepare Smart Rings for open-source collaboration, we hope to make it easier for contributors to add camera integrations, model adapters, evaluation footage, and notification channels.