Easy Tasks AI (E.T.AI)

Training Lobe to understand 'Lights On'
Trying to use Raspberry Pi for Hardware
3D Printed Camera Case For RPI/Jetson-Nano

Inspiration:

We got our inspiration from our grandpa (who is 83 years old) because we noticed a consistent problem of him not being able to turn on the lights of our main room because it was plugged in near the ground, making him bend down. Also, he would often search for the TV remote and struggle to find it. We also have been worried as to how he would communicate in an emergency situation if he wasn't able to speak at that moment. Therefore, we created Easy Tasks AI!

What it does:

We have created a program that uses hand gestures to achieve easy tasks to help the elderly and differently-abled people. Examples of easy tasks include turning the lights on and off, turning the TV on and off, turning the volume up and down, call 911 or emergency contacts during any emergencies, and translating sign language into plain text that anyone can easily understand.

How we built it:

We used a cross-platform machine learning application called "Lobe" to train our model to recognize hand gestures as different commands, and we used "Teachable Machine" to train data for some simple sign language words/ phrases. After that, we used python to create a code that takes a picture every few seconds and uses that to predict a specific output. We also used the IFTTT platform to execute the predicted outputs from the python code. We even created a hardware device using a Jetson Nano and 3D printed some parts.

Challenges we ran into:

Our camera for the Jetson Nano broke so we had to improvise our design plan and ideas last minute which was a big challenge. Also, it was challenging to train the ML model as we were aiming for 98%+ accuracy and after several rounds of fine-tuning, we got there.

Accomplishments that we're proud of:

We were able to get our solution to interpret gestures for some of the really useful tasks. Also, we created a sign language ML which took us a while but we are proud of the outcome predictability we were able to achieve. We are also proud of our python code that predicts the hand gestures and works really well!

What we learned:

We learned what ML is and how it works and also how to use "Lobe" and "Teachable Machines". We learned how to use Jetson Nano, expanded our knowledge about HTML, Python, and IFTTT.

What's next for Easy Tasks AI:

There is no limit! We will expand the tasks that our solution can handle, improve upon the ones that we made so that they become even more accurate and it can help many people in the future. like it is already helping our grandpa!