sidekick

A cooking voice assistant to make cooking easier and more accessible

Comment

Inspiration

Baking during Quarantine + Recipes on Smartphones = Oily, greasy, cakey phone screens

Orange = Yellow + Red (colors yellow and red stimulate hunger the most, plus we like the orange the fruit and the color)

What it does

Sidekick provides an on-command assistant to help make cooking with recipes faster, easier, and cleaner. Sidekick is loaded on your phone, placed in your kitchen, and listens to your commands hands-free after a recipe is selected. Our app verbally walks users through recipes step by step, all while responding to the user’s questions and commands. Sidekick will figure out what you want, and answer you based on your recipe, and will stay with you throughout your whole cooking process. Powered by Facebook's wit.ai NLP API, Sidekick requires absolutely no preset command patterns to respond to audio commands. The app determines the intent behind any question worded in any form and returns the relevant information the user wanted, all hands-free! That means no more greasy screens or misread measurements!

Key Features

  • No preset command patterns, ask questions related to your recipe in any way you want. Users can ask the question "Butter?" or "How much butter?", or "How much butter is required for this recipe?" and receive the same answer ("This recipe calls for 2 sticks of unsalted butter").

  • Sidekick verbally reads recipe instructions to users. Sidekick can be prompted to deliver steps with commands like "go back to the last step", "repeat the current step", and "next" and other variants of similar commands. This makes cooking and recipes easier to follow along for most users, but especially more accessible for those with low vision.

  • Select from a variety of recipes and input your own creations, all on an easy-to-use and accessible interface!

How we built it

Built using Figma, Java, Android Studio, and wit.ai.

Challenges we ran into

Translating our Figma design into an Android app took us time and great attention to detail. We weren't the most experienced with Android Studio, but we learned how to create the layouts and ended up with a final product close to our initial design.

In our backend, getting Sidekick to pick up audio input and translate it into text took us a long time. We had to figure out a way for Sidekick to continuously listen for audio commands when Android Studio doesn't natively support that functionality. Fun fact, emulating Android devices doesn't support audio input. All of us had iPhones, so we had to borrow an Android from a friend for a couple of hours and get our speech recognizer working in that time before we had to return it, which was extremely stressful.

Accomplishments that we're proud of

We're most proud of getting a working application that determines the intent of an audio queue extremely accurately, regardless of how the question is asked.

What's next for Sidekick

We plan to expand Sidekick by incorporating popular recipe websites that users can browse and choose from. The next steps also include integrating popular voice assistants like Alexa/Google Home etc, multiple language support, and social features like recipe sharing and synced audio for social cooking events!

Built With

Share this project:

Updates