Inspiration
Tesla auto driving features, dashcams, predictive analysis, Doorbell and IP cameras all answer the same narrow question — "was there motion?" — not the one that actually matters to a homeowner: "should I care about this?" That gap is why camera apps train people to ignore push alerts entirely. Kenbunshoku ("the art of perceiving," 見聞色) tries to close that gap by adding a layer of visitor context, without ever taking the decision out of the homeowner's hands.
What it does
Any existing doorbell/IP camera streams video → an edge service does local person detection (YOLOv8n) → cropped, motion-triggered frames go to a cloud backend on Alibaba Cloud → Qwen-VL reasons about visitor context (familiar / delivery-like / anomalous, with a plain-language explanation) → a memory store checks whether this is a recurring pattern (same visitor, similar day and time) → the homeowner gets a push notification with the reasoning attached. Recognized, low-priority recurring patterns (e.g. the same delivery courier at the same time each week) are logged but not re-pushed, to cut alert fatigue deliberately rather than silently. The system never takes autonomous action — it informs, the human decides.
How we built it
edge-service/ — Python, YOLOv8n (ultralytics), OpenCV, Docker. Reads frames from any camera stream URL (RTSP/MJPEG/device index — config, not hardcoded), yields person-present detections above a confidence threshold.
cloud-backend/ — FastAPI, deployed on Alibaba Cloud ECS. Calls Qwen-VL (qwen-vl-plus) via the OpenAI-compatible Qwen Cloud API (dashscope-intl.aliyuncs.com), stores visit history in SQLite, decides whether to push based on classification + recognized-pattern context.
notification-client/ — Flutter, listening on an ntfy.sh topic (free tier, zero backend setup). Display-only feed + detail view.
Deployment — Alibaba Cloud ECS, Singapore region, Docker.
Challenges we ran into
An em dash in a push notification's Title header crashed the request outright — HTTP headers must be ASCII/latin-1, and that only showed up once we sent a real alert, not from reading the code. macOS's App Sandbox silently blocked all outbound network requests from the notification client until we added the network.client entitlement; Android needed an explicit INTERNET permission too — neither gap was visible from static analysis, only from actually running the app. We also found that our edge-service's video loop could fall behind a live camera stream during multi-second cloud round-trips, silently processing stale, buffered frames instead of the live scene — fixed with a background frame reader that always keeps the freshest frame.
Accomplishments that we're proud of
Every layer was verified against something real, not just unit-tested: real YOLOv8n detections against a live phone camera stream, a real classification round trip through Qwen-VL, a simulated repeat visitor whose 2nd visit correctly triggered pattern recognition and alert suppression, a real push alert rendered on a physical Android phone, and cloud-backend deployed and verified live on Alibaba Cloud ECS — including catching and fixing four real bugs along the way that would otherwise have shipped silently broken.
What we learned
That "it compiles" and "it analyzes cleanly" are a long way from "it works" — every real bug found this build came from actually running the thing on real hardware against a real network, not from static analysis.
What's next for Project Kenbunshoku
Real OS-level background push (the current notification client holds a foreground connection to ntfy.sh, which is fine for a demo, not for daily use), TLS on the ingestion endpoint, and richer pattern memory beyond the current day-of-week-and-time heuristic. Explicitly not next: weapon/threat detection or any autonomous action — both are deliberate, permanent scope cuts, not gaps to fill later.
Log in or sign up for Devpost to join the conversation.