CAUGHT IN 4K

Catalog of all items

Inspiration

After eating, you stand in front of three bins, one for trash, recycling, and compost, as you struggle to remember which bin to throw your cup into. A couple seconds later, you decide to just shove it in the trash without thinking.

Aggregated across the waste bins at every mall, restaurant, store, and business, approximately 70 - 75% of recyclable/compostable material in the United States is put in the trash.

What it does

Whenever a customer or employee walks towards a set of trash, recycling, and/or compost bins, CAUGHT_IN_4K notices them approaching and tells them where to drop their waste. It uses only existing surveillance cameras and requires only the addition of a speaker near the trash cans.

Businesses or restaurants using our product would get a private dashboard. It shows recent footage, statistics of the items that were placed in each bin, and a catalog of all the types of items that have been recorded in the past.

How we built it

Once connected to existing security cameras, CAUGHT_IN_4K will create skeletons of every person in the room using YOLO11n-pose for pose estimation. From the position of each person’s wrists, it decides whether they’re holding something and crops the item from the frame. We use CLIP ViT-B/32 to classifies these waste products based on existing product labels (paper bag, plastic cup, straw cover, etc.) stored in PostgreSQL. For unknown waste products, Chalk Model Gateway routes the crop to Claude Sonnet 5.5 to propose a product label for review before it is added to the database. Chalk then routes the item description and local disposal rules to TypeSafe Jev, which determines the correct bin. If the model notices that someone is getting close to the trash can, ElevenLabs’s voice API tells the user where to put their trash.

Try it yourself: visit https://caught-in-4k-app.vercel.app/camera#key=16b4451a25e229377d31f471, turn on your camera, hold up any item, and watch the system name it and tell you which bin it goes in. You can also upload a video and get it back with every item labeled.

Challenges we ran into

Our object classification model took multiple iterations to actually work. Initially, we prioritized open-vocabulary detectors like YOLO-World, but they struggled to detect small items held in someone’s hand, especially if the person is facing away from the camera. They also ended up marking items that weren’t in anyone’s hand. To fix this, we used YOLO’s pose detection to find each person’s wrists and start tracking items as soon as they are in the frame of a surveillance camera. Then, we used CLIP to identify the item that is in each person’s hand, marking our first improvement on the model.

Another problem was that the system sometimes detected several items when a person was holding only one. We reduced this by tracking each detection across frames and only confirming an item once it appeared consistently in the same hand for several frames.

Accomplishments that we're proud of

We now have a powerful minimal viable product of CAUGHT_IN_4K which includes a model that accurately tells the users which bin to place each piece of trash in. We also have a fully deployed UI on Vercel that allows prospective clients and existing clients to learn about and manage their Caught_IN_4K products.

What we learned

We learned how to integrate Chalk into agentic processes such as image classification workflow. In this project, we used Chalk to route photos of individual items by cropping around the arm of a YOLO skeleton to Claude and then to route the item description and relevant disposal rules to Jev in order to select an appropriate bin.

What's next for CAUGHT IN 4K

Our plan for the project dreams big, we plan on:

Running more validation tests in various live environments.
Connecting the product to a standard surveillance camera to see how effective it is.
Build a hardware version of the product that only opens the bin the waste product belongs in.
Offering free subscriptions to local businesses to begin developing a clientele.