Shopiva - Grocery Shopping for the Visually Impaired

Inspiration

Stop and imagine: what would it be like to shop in a grocery store, blindfolded? The loud yelling, persistent machine beeps, and the faint smell of paint might start to overwhelm you as you search around the entrance to find a store assistant to shop with.

Unfortunately, this is likely a bi-weekly occurrence for 39 million people who are completely blind and 244 million others with some form of visual impairment.

Despite the growth of grocery delivery apps during COVID-19, people with visual impairments have been left behind. From our user research of the blind community, we discovered that existing grocery shopping apps have small and hard-to-adjust text, no or poor auditory interfaces, and are generally not designed with the visually impaired in mind.

Our team of three set out to tackle the problem of: how can we empower people who are visually impaired to get their essentials?

What it does

With the existing problems in mind and an accessibility-first approach, we designed Shopiva, a grocery shopping app to empower the visually impaired featuring an accessibility-centric interface design and a voice-only mode.

Every click has audio feedback to guide people with impaired vision, and the large icons and text size provide high readability. Floating plus and minus buttons are present on every screen to easily adjust the content's size. In addition, all of the colour contrasts in the app were designed to meet the double A web content accessibility guidelines to support colour-blind users.

To make searching less dependent on eyesight, we can search for an item using voice with Google's Speech-to-Text API. On every page, a floating microphone button is shown that we can press down on to search by voice. We also built the ability to search by image. Using Google's Cloud Vision API, we identify the object in the photo, and use it to query our grocery database.

Despite this app already being a significant improvement for those with less severe visual impairments, over 44 million people live in complete blindness and require a radically different solution than an accessible app. Additionally, we built a voice-only experience to provide the same richness and control provided by our app but without the visual barriers.

How I built it

We built our app using React Native and TypeScript to support both iOS and Android users. We integrated with the phone's camera and microphone hardware to collect auditory and visual inputs for simpler searching. Our backend is written with Express and node.js and hosted on the Google Compute Engine, and we leveraged Google's Speech-to-Text and Vision APIs for natural language processing and computer vision. We built our voice-only mode using VoiceFlow and Google Assistant.

Challenges I ran into

This hackathon was Lily's first time designing an accessibility-first app. Design for accessibility was extremely challenging for this project, and we had to have an unwillingness to compromise on accessibility for aesthetics in order to solve the root problem. Selecting contrast-friendly colours and designing the app's screens to support multiple text sizes was particularly challenging.

This hackathon was Derek's first time working on an app, and he faced challenges gaining familiarity with the development environment and integrating with the native hardware modules.

This hackathon was Antonio's first time independently building an integration with Google's Vision API, which was a great learning experience.

Accomplishments that I'm proud of

We're very proud of building two complete products providing very different (yet both important) user experiences - both the visual experience and the voice-only experience.

What I learned

Accessibility design is hard, but should be a primary consideration rather than an afterthought. The people who can benefit from our support as hackers the most are often the most neglected when profit-based motives arise, and as the next generation of engineers and designers, we can make an effort to change that.

What's next for Shopiva - Grocery Shopping for the Visually Impaired

Improve the aesthetics of the UI while maintaining accessibility standards
Conduct extensive usability testing with people with a variety of degrees of visual impairment