nkiruka

Inspiration

Sometimes we can't always ask someone to take pictures for us, and selfies might not be the best option, so we wanted to create an app that allows users to take pictures freely without the need to ask someone else or worry about bad angles.

What it does

This is an app that recognizes keywords via voice recognition model and takes a picture on the mobile device for users.

How we built it

We trained the deep learning model using PyTorch libraries and built-in speech commands in torchaudio datasets. The app development was programmed in Javascript using Android libraries.

Challenges we ran into

During app development, we had problems with continuously sending data from microphone to the model for voice recognition. The app crashed and showed an "unknown" error. After model training, we serialized the model to be used in the app. However, Pytorch Mobile only works with Pytorch version 3 and our model was developed in Pytorch version 2.

Accomplishments that we're proud of

This is a team of 3 members, we were not able to meet together to discuss and work together on this project due to the covid situation. However, we managed to communicate effectively online and worked remotely together on this project. Meanwhile, all team members have jobs and other responsibilities outside of this project.

What we learned

During this project development, we learned to be responsible for our tasks and to update the team on any changes coming up. We learned some aspects of software project management that we have not learned before during this covid situation.

What's next for nkiruka

Next, we hope to fix the Pytorch version issue so that the app can use the voice recognition model. After fully implementing the english version model, we hope to include voice recognition models in other languages. Eventually, we hope to include gesture recognition for those who have speaking disabilities and for situations when voice recognition is not ideal.

Built With

Submitted to

PyTorch Summer Hackathon 2020

Created by

I worked on the Android Application, building utility functions and organizing the architecture of the application.

Tristan Hilbert
I worked on the deep learning voice recognition model. It is my first time training in PyTorch and working with audio data. We developed a model that has great performance, but PyTorch Mobile only works with pytorch version 3 and our model was in pytorch version 2, so we couldn't integrate the model in the app. We will continue working on this project to fully implement all the functionalities.

Jayee Li
Computer Science, MS at Tsinghua University
lyubogankov

Updates

Jayee Li started this project — Aug 25, 2020 02:35 PM EDT

Leave feedback in the comments!

Log in or sign up for Devpost to join the conversation.