Inspiration
Online shopping has become increasingly accessible through screen readers, voice assistants, and mobile accessibility features. But there is still a major gap after an order reaches the customer.
A blind or visually impaired person may be able to place an order independently, yet when the package arrives, identifying the exact physical product can still depend on sight or help from another person.
A product may have the correct brand and packaging but the wrong variant, size, or quantity. For example, someone may order an Aloe Vera lotion but receive a Vitamin C variant instead. If the user already knows they need to avoid a particular ingredient or variant because of an allergy or sensitivity, independently identifying what was actually delivered becomes especially important.
That inspired OrderCheck AI: a voice-first delivery verification assistant designed to help blind users understand exactly what arrived without depending on another person.
What it does
OrderCheck AI lets users describe their expected order naturally using their voice.
For example:
“I ordered two Vaseline Healthy Bright Perfect 10 Pro-Age Repair Lotions, 200 millilitres.”
The application transcribes the order, extracts important information such as product name, brand, variant, size, and quantity, and reads it back for confirmation.
When the delivery arrives, the user points their phone camera toward each product. OrderCheck detects the barcode, searches the product catalog, and retrieves the exact product information.
It then compares the ordered product with the received product and provides clear spoken feedback such as:
- Match
- Mismatch
- Unable to verify
- 1 of 2 received
- 1 item still missing
If the barcode cannot identify the product, OrderCheck can use OCR as a fallback to read information from the label.
The goal is not simply to identify a barcode. It is to answer a more useful question:
“Did I actually receive what I ordered?”
OrderCheck does not diagnose allergies or determine whether a product is medically safe. Instead, it helps users identify the exact variant they are holding so they can make their own informed decisions.
How we built it
OrderCheck AI was built as a responsive web application using Next.js, React, and TypeScript.
For voice input, we use Groq with Whisper transcription to convert natural spoken orders into text.
The application then structures that information into fields such as:
- Brand
- Product name
- Variant
- Size
- Quantity
For physical product identification, we use ZXing for barcode scanning.
The barcode is matched against a product catalog stored in Supabase PostgreSQL, which contains structured product information including barcode, brand, name, variant, and size.
For products that cannot be identified through the barcode, Tesseract.js OCR provides an additional fallback for reading product labels.
A custom verification engine compares the expected order against the physical evidence and determines whether the product matches, whether there is a conflicting variant, and how many units have been received.
The application also includes spoken accessibility feedback and supports a voice-first workflow while keeping buttons and keyboard controls available.
We use GitHub for source control and collaboration and Vercel for deployment.
The project also includes unit tests, browser tests, end-to-end workflow tests, and accessibility-related testing.
Challenges we ran into
One of the biggest challenges was making real-world barcode scanning reliable.
Barcodes are not always perfectly positioned. They may be printed vertically, wrapped around curved bottles, partially visible, or affected by lighting and camera angle.
We improved the scanner by supporting common retail formats such as EAN and UPC and adding additional scanning strategies for difficult barcode orientations.
Another major challenge was exact variant matching.
In one test, the user said they ordered an Aloe Vera version of a lotion, but the received product was Vitamin C Whitening. Initially, because the parser had included “Aloe Vera” as part of the product name instead of treating it as a separate variant, the system incorrectly considered the product a match.
We updated the verification logic so that meaningful descriptors left in the spoken product name are also checked against the database variant. This allowed the application to correctly report the mismatch.
Quantity tracking was another important challenge. A single successful scan should represent one physical unit, not automatically complete an order of multiple identical items.
We also had to make sure incorrect scans never consumed the expected quantity.
Another challenge was designing spoken feedback that was detailed enough to help a blind user without becoming confusing or repetitive.
Accomplishments that we're proud of
We are especially proud that OrderCheck AI works with real physical products, not just simulated data.
The application can:
- Accept natural voice-described orders
- Transcribe and structure them
- Scan real product barcodes
- Retrieve exact product information from a cloud database
- Detect wrong variants
- Track quantities across multiple scans
- Report missing products
- Use OCR as a fallback
- Speak verification results to the user
We are also proud of the exact variant comparison.
OrderCheck does not simply say:
“This is body lotion.”
It can tell the user:
“You ordered Aloe Vera Body Lotion. You received Vitamin C Whitening Body Lotion. This product does not match your order.”
That small difference can be very meaningful for someone who cannot simply read the label visually.
Another accomplishment is building the complete flow as a deployed working application rather than only presenting a concept.
What we learned
The biggest lesson we learned is that accessibility is not only about making a screen readable.
A user may have an accessible website, an accessible checkout process, and a screen reader, but accessibility can still break down when they have to interact with a physical object.
We also learned that confidence matters.
If OrderCheck does not have enough reliable evidence, it should say “Unable to verify” rather than inventing an answer.
For an accessibility-focused application, an honest uncertain result is better than confidently giving incorrect information.
We also learned how important structured product data is. Barcode scanning alone gives us an identifier, but combining it with a product database makes it possible to understand the exact brand, variant, and size.
Finally, testing with physical products showed us how different real-world conditions are from ideal software tests. Camera angle, curved packaging, lighting, barcode placement, product variants, and natural speech all introduce challenges that only appear when the complete system is used end to end.
What's next for OrderCheck AI
The next step is to expand OrderCheck beyond a hackathon MVP.
We would like to:
- Expand the product catalog to support many more products and retailers
- Integrate directly with e-commerce platforms so users do not need to manually describe every order
- Improve OCR and label understanding for unknown products
- Add better camera guidance for users who cannot visually position the barcode
- Support more languages
- Test the application with a larger group of blind and visually impaired users
- Improve recognition of ingredients and product attributes while clearly separating identification from medical advice
- Explore manufacturer and retailer product-data integrations for more accurate and scalable product information
Our long-term goal is simple:
A person should not need sight, or another person, just to know whether the correct product arrived.
Built With
- accessibility
- axe
- barcode-scanning
- computer-vision
- github
- groq
- next.js
- ocr
- playwright
- postgresql
- pwa
- react
- responsive-design
- rest-api
- speech-recognition
- supabase
- tesseract.js
- text-to-speech
- typescript
- vercel
- vitest
- voice-ui
- web-speech-api
- whisper
- zxing
