Mira | Devpost

Inspiration

Grocery shopping is full of small decisions: finding the right aisle, choosing a familiar brand, checking a label, and remembering what is already in the cart. For blind and low-vision shoppers, much of the information needed to make those decisions is presented visually.

We wanted to make that information accessible while keeping the shopper in control. That meant thinking beyond a camera app that requires someone to hold up a phone and aim it at every product. Our idea became Mira: a shopping companion that brings together conversation, computer vision, and a cart-mounted phone, with the goal of making everyday shopping more independent.

What it does

How we built it

Mira lets shoppers build and manage a grocery list through conversation. They can ask for specific products, brands, sizes, and quantities, or bring back familiar items from previous trips.

During a shopping trip, Mira plans a route through a mapped store and provides spoken directions to the items on the list. At each stop, the camera reads packaging or recognizes produce to help identify the requested product. Mira asks for confirmation before checking an item off, keeping the shopper involved in the decision. Completed lists are saved in History for future trips.

We are also developing a motorized phone mount and Apple Watch feedback to make the experience more hands-free. The mount is designed to position the camera, while wrist vibrations communicate directions and obstacle alerts.

Challenges we ran into

Some of the things that we struggled with included using the voice agent for navigation, connecting to the hardware arm. It was most of our first times working with swift/xcode, and therefore we would push each others environment variables and settings into each others environment. Another thing we struggled with was building the navigation map using a calibration system, we spent about a total of 5 hours at target perfecting the math and placing nodes at every part of the map.

Accomplishments that we're proud of

user can successfully conversate with Mira to add items to the grocery list, the grocery list then maps back to the prodsuct catalog which was scraped directly using the redsky API. We successfully integrated eleven maps voive agent to the application and we used Speech to text and Word extraction.