Inspiration
Halloween is coming up, and every "please take one" bowl has the same problem: someone takes six. We wanted a candy box that makes sure every kid gets a fair share, and is funny about it.
What it does
A camera watches the candy box. Take one piece and the plushie guarding it looks pleased. Take two and it warns you. Take three, or leave your hand in there, and the cardboard lid shuts while the plushie roasts your costume out loud: "not you and your cheap dollar store mask trying to get extra candy."
Come back later for seconds and it recognizes you by your outfit.
The TreatOps dashboard shows the live camera, a "Top Big Backs" leaderboard with each offender's photo and candy count, and a replay of every roast. From it you can close the lid, trigger a roast, or hold a button and talk through the plushie yourself.
How we built it
- Hands: MediaPipe's hand-tracking model sees when a hand goes into the box and when it leaves.
- Box position: an ML model finds the candy box in the camera's view, so the box can move while the camera stays put.
- Counting: an OpenAI model compares photos of the box from before and after each grab, plus the clearest frame of the hand leaving, to tell how many pieces were taken.
- Repeat visitors: a clothing recognition model turns each visitor into an embedding, and a vector search against everyone who came by tonight catches anyone coming back for seconds.
- Roasts: OpenAI writes a short, brainrot-style roast from a photo of the costume, and ElevenLabs voices it. We start writing the roast the moment a hand goes in, so it's ready when the lid shuts.
- Plushie: a Freenove ESP32 with a 2.8" screen for its eyes and a speaker. The laptop sends each line to it over Wi-Fi. The eyes follow your hand, change mood, and move with the voice.
- Lid: a servo on an Arduino Uno, which the laptop tells to open or close.
- Dashboard: a web page served by the Python app. The push-to-talk audio is recorded in the browser and played on the plushie.
Challenges we ran into
- Finding a motor strong enough to hold the box open, then calibrating the servo so the lid closed all the way without swinging too far open.
- Counting candy from one photo was unreliable. Comparing before and after photos of the box fixed it.
- The AI calls take about 6 seconds, which kills a joke. We hid the wait by starting them as soon as a hand reaches in.
- The plushie's speaker stayed silent until we found that mono audio only drove the ESP32 pin the speaker wasn't on.
Accomplishments that we're proud of
- It works end to end: grab, count, roast and lid in a few seconds.
- The roasts are actually about your costume, not generic insults.
- It remembers who already came by, so going back for seconds doesn't work.
- The dashboard makes it feel like a real security operation.
What we learned
- Timing is the whole joke. A roast that lands late isn't funny.
- Comparing before and after beats guessing from a single frame.
- Hardware fights back: pin conflicts, servo angles, motor strength.
What's next for TreatOps
- Weigh the candy with a load cell as a second check on the camera.
Log in or sign up for Devpost to join the conversation.