TOMMY

Inspiration

Our Inspiration stems from one of our members brothers that has Cerebral Palsy. While they posses what some would consider a mild case, they still struggle with using their hands to type and use a traditional mouse. That's where T.O.M.M.Y. comes into play. With T.O.M.M.Y. we can give people with certain disabilities or people that just want an extra hand the ability to use their internet browser in a more efficient and comfortable manner.

What it does

The "Talk Operated Multi-Media System" or TOMMY for short is a Speech-to-Text system that uses OpenAI to allow users to navigate the web all with just their voice. Not only that, TOMMY can also summarize pages.

How we built it

Using React, Typescript, OpenAI, and Manifest 3 we are able to convert Speech-to-Text into commands that navigate your browser.

Challenges we ran into

The biggest issue we've faced by far is having Chrome communicate with the front end. One of the side effects of that was not being able to get the SVG logo to load onto the chrome extension.

Accomplishments that we're proud of

Some accomplishments that we're proud of include injecting an extension into the DOM. We were able to get done multiple ways for the AI to interact with the browser. As well as build a communication system between the fronted and the extension. But the biggest one of all was being able to filter useful information from the webpage.

What we learned

We learned that chrome extensions are different than we initially thought and that they can have their own sets of issues that you need to be prepared for.

What's next for TOMMY

Some plans for TOMMY is that we'd like to go back into the code base and look at some of the code that could either be better written or optimized that we had to graze over in the sake of time. Another thing would be adding more functionality and working on the front end look. This would include more themes and potentially more customization options.

Built With

• openai
• react
• typescript