Jaimes AI

Jaimes is an AI agent meant to guide the visually impaired through the web. Using speech to text and text to speech, users can automate their tasks on the web in a way that is smooth and easy.

Comment

Inspiration

In hopes of giving those who are visually impaired an easier time on the internet, we opted to make an application to do just that!

What it does

Jaimes AI is a Chrome extension that allows users to explore the web. Users may enter text into inputs through voice, navigate through links with voice, and receive audio feedback on what buttons exist on the page.

How we built it

Using Google Cloud APIs to process speech to text and text to speech, we passed data through Meta’s Llama 3.2 model to categorize user prompts into actions for the client to process. Our Chrome extension calls to our Flask API, which then interacts with the LLM processed by Nosana’s GPU service and hosted on Cloudflare.

Challenges we ran into

We ran into a few formidable obstacles that we needed to overcome. The main one was a sheer lack of manpower, considering the scope of the app that we were creating. Another obstacle that we faced was the difficulty with setting up connections between a remote jupyter notebook, something that perplexed us for a bit. Finally, tackling the Google Extensions logic was also no small feat, and took us a while to soundly implement.

Accomplishments that we're proud of

We are proud of being able to design a system that allowed us to solve a real problem. Being new to the fields of artificial intelligence, we faced many new and unfamiliar challenges, and despite maybe not being able to create the most perfect product, we are glad to have accomplished so much in such little time. We are also proud to have taken on a large variety of different technologies, from complex internet routes to audio formatting and retrieval.

What we learned

The biggest thing that we took away from this event was how important planning and pacing is. Considering the scope of our project, there were times when we got too comfortable, and ended up progressing slower than we should have. This was especially notable within the last six hours of development, when the sheer amount of burnout caused us to progress much slower than before.

What's next for Jaimes AI

Our hope is that we can develop Jaimes to one day be a fully capable AI agent, who wouldn’t only provide assistance to the visually impaired, but also to the everyday man. We hope to do this through taking data from everyday users and fine tuning our models so that they can better assist those in need.

Share this project:

Updates