the demo page with “No backpacks or bags. No jeans.”
a close-up of the plaza detection. Bags show red “Handbag/Backpack in view”, and people show green “Jeans: no” or amber “Jeans: yes”.
the construction example with one worker marked “Hard hat: not seen” next to a fully geared one, plus the rule caption.

Inspiration

Cameras are everywhere, but nobody watches every feed. HawkWatch (TreeHacks 2025) showed that Gemini could turn any camera into a security system. We wanted the rules to come from the people who run the space, so anyone can type "No backpacks or bags" or "No jeans" and the camera starts checking for it.

What we learned

We were honestly shocked by how much a camera can pick up now. On a crowded plaza clip, an off-the-shelf detector boxed around 19 people per frame and followed each of them, along with every backpack and handbag, and Gemini could tell from a small crop whether someone was wearing jeans. None of it needed custom training.

How we built it

YOLO-World and ByteTrack find and track the people and bags in each frame. Gemini 2.5 Flash turns the typed policy into structured rules, and Pydantic checks them against a small fixed vocabulary. A Python rule engine applies the rules on every frame, with timers and one alert per person. It sends only selected crops to Gemini for yes/no questions like "is this person wearing jeans?". FastAPI streams the annotated video to a React page, and a new policy swaps into the running camera without a restart.

Challenges we ran into

On a laptop CPU the detector managed about 1.5 fps on a crowded plaza, so we recorded the tracks ahead of time and cached Gemini's answer for each person. A cold Gemini call also took about 11 seconds until we switched to a regional endpoint, which brought it to under 4.