#14 - Auto-Mate

Title and App Screen

Inspiration

Our inspiration for this project was our grandparents. They struggle to keep up with new advancements and have difficulties performing what might seem like mundane tasks to us. However, with the app we're building, we take the effort out of something like booking an Uber, so that they, too can have access to a technology-driven world without much learning and effort to do.

What it does

Our phone app prototype uses LLMs to parse the screen, learn what the app is currently trying to do, and proceed to the next step in achieving a goal. Controlled by simple English commands, the LLM breaks it down into several steps and executes them. In the future, we want to add a login option and a favorites bar, so that presets can be used to make it much easier.

How we built it

The back end was built using Python. For the app, we used react native and Expo Go on an Android emulator to develop the app and ensure it connects and stays in contact with the LLM.

Challenges we ran into

A significant challenge that we have yet to fix, is the voice-to-text functionality. As a fledgling development tool, Expo Go still does not support speech-to-text. React-Native workarounds are also not reliable and have to be debugged to solve completely. Apart from this, using LLMs to control Android devices is quite novel, and not much prior work exists. As a result, we had to develop a lot of the methods and connections ourselves (and it still doesn't work, unfortunately)

Accomplishments that we're proud of

This is a quite novel idea, with no predecessor in the app market apart from assistants like Siri and Google that have limited cross-app functionality. With this in mind, we are very proud of the fact that it works as intended, even if it is just for one app as a proof of concept. We believe this app has the potential to go very far in the accessibility space.