Inspiration
What it does
How we built it
Challenges we ran into
Accomplishments that we're proud of
What we learned
What's next for FridgeCheck
Inspiration
Anyone who has lived in a dorm, a shared apartment, or worked in an office kitchen knows the feeling: you open the fridge, and the leftovers you were looking forward to are gone. Nobody knows who took them, and nobody is admitting anything.
We built Fridge Tracker for the Most Over-Engineered track. We took this small, universal annoyance and asked what it would look like to solve it with an unreasonably sophisticated system: computer vision, event streaming, and a full dashboard, all to answer one question: who took my food?
What it does
Fridge Tracker watches a shared refrigerator and keeps an automatic record of what goes in, what comes out, and who did it.
- Detects, identifies, and tracks the people interacting with the fridge.
- Recognizes food items and tells apart placing an item inside, removing it, and simply touching or moving it.
- Associates each food interaction with a person and an item, updates the inventory, and checks for ownership conflicts.
- Provides a web dashboard for inventory, registered users, event history, and ownership alerts.
- Exposes fridge data to compatible AI assistants through a read-only MCP server.
How we built it
1. Facial Recognition Pipeline
Our first pipeline uses AI models to detect, identify, and track people at the refrigerator. The architecture explores:
- SCRFD and YOLO-face for face detection
- ArcFace and AdaFace for facial recognition
- BoT-SORT for tracking individuals across video frames
- Conformal prediction for handling uncertain identities and distinguishing registered users from unknown people
These techniques are intended to improve reliability under challenging conditions such as poor lighting, unusual camera angles, and partial occlusion.
2. Food Detection and Object Tracking
The second pipeline focuses on understanding what happens inside the refrigerator. We explored:
- YOLO and Grounding DINO for object detection
- SAM 2 for object segmentation and tracking
- MediaPipe Hands and HaMeR for hand tracking and interaction analysis
- DINOv2 for visual item identification
- The Hungarian algorithm for matching detected objects with previously registered inventory items
The goal is to distinguish between placing an item inside, removing it, and simply touching or moving it.
3. Backend and Event Processing
We designed an event-driven backend to combine information from both camera pipelines:
- Python and FastAPI for backend services
- Redpanda (Kafka-compatible) for event streaming
- PostgreSQL for storing events and inventory information
- Event-time window joins to associate detected people with food interactions
- Event sourcing and CQRS to reconstruct inventory from recorded events
When the system detects a food interaction, the backend associates the action with a person and an item, updates the inventory, and checks for ownership conflicts.
4. Web Dashboard and Infrastructure
We designed a React-based dashboard for monitoring the refrigerator. It provides access to inventory, registered users, event history, and ownership alerts.
For infrastructure, we explored Docker, Kubernetes, autoscaling, and observability tools such as Prometheus and Grafana. We also designed a read-only MCP server to make refrigerator data accessible to compatible AI assistants.
Challenges we ran into
- Synchronizing people and objects: Recognizing a face and detecting a food interaction are separate processes. We needed a way to associate events accurately, even when camera streams and processing times differed.
- Handling imperfect recognition: Lighting changes, occluded objects, similar-looking food packages, and unfamiliar faces can all produce incorrect predictions. We explored model ensembles, tracking, and confidence-based decisions to improve reliability.
- Running computationally expensive AI models: Many advanced computer vision models are optimized for NVIDIA GPUs, while our available hardware was based on Apple Silicon. This created compatibility, memory, and performance constraints.
- Balancing complexity with functionality: Since we were competing in the Most Over-Engineered track, we wanted to explore advanced technologies without losing sight of the actual problem. Every additional component needed to serve a meaningful purpose.
What we learned
This project gave us the opportunity to explore how multiple areas of software engineering and artificial intelligence can work together. We learned about:
- Combining computer vision models into a unified pipeline
- Designing event-driven architectures and distributed systems
- Managing asynchronous data from multiple sources
- Working with facial embeddings, object tracking, and confidence scores
- Understanding the trade-offs between AI accuracy, processing speed, and hardware limitations
- Collaborating on a project with independent services and shared data contracts
Most importantly, we learned that building a sophisticated system isn't just about adding more technology. It's about understanding why each component exists and how it contributes to the final product.
What's next for Fridge Tracker
We see Fridge Tracker as an interesting foundation for smarter shared spaces. Future improvements could include more robust object recognition, improved performance on edge devices, and support for deployment in dormitories, shared apartments, and office kitchens.
Ultimately, our vision is simple:
A world where your food can disappear from the fridge, but the mystery of who took it doesn't have to remain unsolved.
Built With
- adaface
- apple-silicon
- arcface
- bot-sort
- computer-vision
- cqrs
- dinov2
- docker
- event-sourcing
- fastapi
- grafana
- grounding-dino
- hamer
- kafka
- kubernetes
- mcp
- mediapipe
- postgresql
- prometheus
- python
- react
- redpanda
- sam-2
- scrfd
- yolo