Inspiration
The inspiration for this project came from the growing demand for contactless and hygienic food ordering systems, especially after the COVID-19 pandemic. We wanted to design a system that makes ordering food intuitive, safe, and futuristic by replacing traditional touchscreens with hand gesture recognition. The idea was to bring together Computer Vision (CV), Artificial Intelligence (AI), and User Interface Design (UI) to create a seamless human–machine interaction experience.
What it does
The Gesture-Controlled Smart Food Ordering System allows users to order food without touching any screen. Using only hand gestures detected by a webcam, the user can: Swipe left/right using thumb movements to navigate through menu items. Thumbs up to add the selected item to the cart. Thumbs down to open the cart page. On the Cart Page, another thumbs down navigates to the Payment Page. Finally, a thumbs up confirms the order and completes the transaction. It provides a fully hands-free, AI-powered ordering experience, perfect for public kiosks, restaurants, or smart cafés.
How we built it
Gesture Recognition
We used MediaPipe Hands to detect and track 21 hand landmarks. OpenCV processed the webcam feed in real-time. User Interface (UI): Built using Tkinter for simplicity and flexibility. Each product (e.g., burger, pizza, dessert) has a custom background with premium silver and gold tones. The camera feed is overlaid dynamically on top of the UI to guide the user visually. System Flow: Menu → Cart → Payment pages transition automatically based on gestures. Cart and total amount are dynamically calculated. GUI updates are handled within the main Tkinter loop for real-time responsiveness.
Challenges we ran into
Gesture Misclassification: Initially, the system struggled to differentiate between thumbs up and thumbs right due to lighting and background interference. We fixed this by adding geometric angle thresholds and smoothing filters. Frame Lag: Balancing real-time video capture with GUI rendering was challenging. We optimized the frame rate by updating only selective UI elements instead of redrawing everything per frame. Lighting Variations: Gesture detection accuracy dropped in low-light conditions. We resolved this using adaptive contrast adjustment in OpenCV. Overlapping GUI Elements: Ensuring that the camera preview, background image, and text overlays aligned properly took several iterations.
Accomplishments that we're proud of
Successfully developed a fully gesture-controlled ordering system with no need for mouse or touch. Created a modern, visually appealing UI with individual backgrounds for each product. Integrated real-time camera feed and gesture tracking within a responsive Tkinter interface. Built a smooth navigation flow between menu, cart, and payment screens using intuitive hand gestures. Designed a project that is not only functional but also practical and scalable for real-world deployment.
What we learned
How to integrate MediaPipe Hands and OpenCV for robust real-time gesture detection. How to synchronize Tkinter’s GUI mainloop with continuous video capture without freezing the interface. The importance of user experience (UX) design in accessibility-focused systems.
What's next for Gesture-Controlled Smart Food Ordering System
💬 Voice Command Integration — Add voice recognition for hybrid gesture + speech control. ☁️ Database Connectivity — Store orders, manage menus, and track analytics. 👥 Multi-user Recognition — Support for detecting multiple customers using face recognition. 🤖 Machine Learning Gestures — Train a custom model for personalized gestures. 📱 Deployment on Kiosks and Raspberry Pi — Transform it into a real-world smart restaurant system.
Log in or sign up for Devpost to join the conversation.