OrderLens

OrderLens Architecture

Inspiration

In fast-food restaurants, mistakes often happen between the POS order and the actual food tray given to customers. During peak hours, staff work under time pressure, which leads to wrong items, missing items, or incorrect quantities. These mistakes cause customer complaints, food waste, and slower service.

We wanted to build a system where AI acts as a final verification step before the tray is handed to the customer. Instead of replacing the cashier, the AI checks whether the tray matches the POS order and alerts staff if something is wrong.

This idea came from observing how many operations in restaurants are still manual and error-prone, and we saw an opportunity to apply computer vision and generative AI to solve a real operational problem.

What it does

OrderLens is an AI-powered tray recognition system that verifies whether the food items on a tray match the POS order.

The workflow is simple:

The cashier creates an order in the POS system (items and quantities).
A photo of the food tray is captured using a webcam or uploaded image.
AI analyzes the image and detects the items and their quantities.
The system compares detected items with the POS order.
The system reports whether the tray matches the order or if there are missing, extra, or incorrect items.

This helps reduce order mistakes, improve service accuracy, and reduce customer complaints.

How we built it

We built the MVP with a simple and scalable architecture.

The frontend is built with Next.js and provides:

POS order builder interface
Tray image upload or webcam capture
Verification result dashboard

We used a Next.js API route as a lightweight backend layer to:

Receive the order and tray image
Build prompts for the AI model
Call Google Gemini Vision API
Return structured JSON results

The AI model analyzes the tray image, detects items based on a fixed menu, and returns detected items and quantities. The system then compares the detected items with the expected order and determines whether the tray matches the order.

Challenges we ran into

One of the biggest challenges was making AI reliably detect items from tray images. Lighting conditions, camera angles, and packaging similarities can affect detection accuracy.

Another challenge was designing structured AI output so the system could automatically compare detected items with the POS order instead of relying on free-text AI responses.

We also had to carefully design the demo environment to ensure consistent image capture and reliable results for the hackathon demonstration.