Inspiration
One of our team member has a daily problem that he always forgets to bring his stuff before he leaves his house, and it causes lots of trouble for him. Therefore, he came up an idea that what if an application can remind him when he forgets something. We, as friends, gave birth to this Morning Sanity Checker application so that he won't suffer again.
What it does
This is a Morning Sanity Checker(nickname: Misciker)! It utilizes Google Clouds Voice APIs and Map Matrix API to inform you of your morning commute time to work and prevents you from forgetting your daily items that you take for that day.
It allows you use voice control to set up the device. When users first time use the application, they need to create a morning item checklist. Once everything is set, users can talk to Misciker.
For example, if users say, "Good morning Misciker", it will notify users the current time, and the travel time to work for that day, accounting for incidents along the route, which the route is a route on Google Map between their current location and work location.
Another example, assume a checklist has "wallet, phone, key" as its items. If an user say, "Hello google, I'm leaving home right now, I have my phone and key with me, am I missing anything?" Misciker will then tell the user that he/she forgot to bring his/her phone.
All responses are made by voices. Of course there more features, so let's try it out!
How we built it
By integrating Google Cloud Text to Speech, Speech to Text, and Distance Matrix APIs, we gave Misciker abilities to listen from users, talk to users and predict traveling time for users. The main goal is to build an application remind users when they forget anything before leaving home. There is a keyword_finder to analyze the speech information from users. Here's the core flow of the application. There are three main parts, Speech to Text converter, keyword_finder and Text to Speech converter. The application takes users' voices(e.g. "... I have my phone and key with me ...") as inputs, then uses Speech to Text API convert the speech inputs to texts which are fed to the keyword_finder. The keyword_finder returns a list of items that the users forgot(not included in the input voices). The forgotten_item_list is converted to voices by Text to Speech API, which then the users can hear the reminds.
We finalized this core implementation on Saturday morning after facing challenges on Friday night. Then we assigned tasks for each team member. One person was in charge of keyword_finder, one team member focused on application configuration, one person was responsible for integration and the last person was developing other cool features. After lunch, we had the core running. During Saturday afternoon, we finished our program starting configuration part. We added travel time prediction feature after dinner. We used most of the night time for testing and debugging, and improved some features such as allow some parts of configuration use voice control.
Challenges we ran into
Our original plan was use Google Cloud Vision API to create the check list reminder; however, the build-in camera of the laptop didn't have a decent performance for taking photo of objects those are on the table. Then we tried to build an IOS app instead, but none of us know how to program in Swift. At the end of the day, we have to think out-side-of the box. The goal of the project is to remind users if they forget anything before leaving home in the morning, so instead of showing an object to the device, users could tell the device what they are bringing. The alternative has the same job done, and it seems to be more doable; therefore, we solved our problems by using a different approach. Also, when we were developing more and more features, we ran into issues such as the code worked as expected sometime, but sometime it didn't. We wrote a lot of test cases to minimize the problem, and even applied several stress tests by using the voice recognition feature while simulation noises in the background.
Accomplishments that I'm proud of
I'm proud of we as a team of four can leverage technologies(google cloud APIs) that we don't have many experiences with in a short amount time, and deploy a powerful application that may solve day-to-day problems. In addition, I'm proud of my teammates who didn't give up searching for solutions when they have absolutely no idea how to implement some cool features, but they figured them out eventually. Last, of course I'm proud of myself integrating different APIs into one application.
What we learned
We learned about how to use Google Cloud APIs, and how to communicate with my teammates so that they can have a clear understanding of what each individual is doing. Also, the importance of gathering ideas from different people as a team. We wouldn't be able to create our Misciker without brainstorms from team members.
What's next for Morning Sanity Checker
Deploy the application on a Jetson Nano(with a camera installed) to make a smart home device. And develop an object detector and text recognizer by using the camera on board as eyes, such that people who aren't able to speak could use the application as well.
Log in or sign up for Devpost to join the conversation.