Webly

• 

  Webly landing page

• 

  tailored Hume.ai voice interface

Inspiration

We noticed that the seniors and older adults around us struggled significantly with using online platforms. Most did not even bother using the Internet since they could not figure out how to use it. This led to increased social isolation and an inability to access the opportunities available online. Our grandparents, for example, had to keep asking us how to complete basic Internet tasks. We also wanted to reduce the necessity to type since many older adults struggle with fine motor skills. With AI gaining more functionality each day, we thought about using AI to make searching and using the Internet easier.

What it does

Webly is a platform where people can learn how to use the Internet. The user has to click a singular button and will be directed to a voice interface where they can ask questions about navigating the Internet. For example, the user can ask "how do I write an email" , and the voice interface will vocally guide them step by step on what to click.

How we built it

We built the base using Reflex, an open-source framework to build and deploy web apps. We integrated Hume.ai onto the website, a voice interface that we connected to Google's Gemini that would answer the users' questions. We spent time tailoring Hume and prompt engineering Gemini to answer questions. We also developed a spacy NLP algorithm to analyze the call logs and record all the skills the user has developed through interacting with the voice interface (ex: using Gmail to write an email).

Challenges we ran into

Initially, finalizing our idea was difficult since we needed to make sure it was feasible in the given time period. Integrating AI platforms that have not been integrated before and have different system requirements was a challenge. We had to experiment greatly in terms of accessing and implementing the APIs. Building with Reflex was a challenge since we had to edit our HTML skills to coordinate with the AI platform and had to learn how the platform worked from scratch.

Accomplishments that we're proud of

We're proud that we were able to build with Reflex despite never using the platform before. We also successfully prompt engineered Gemini to guide users step-by-step, rather than all at once like LLMs usually do. Since wanted our app to be accessible, we had to remove the need to type as much as possible. Implementing voice was much more difficult than creating a text-based chatbot but we were able to make it work.

What we learned

We learned how to integrate code into AI platforms. Specifically, We translated Python code developed in a research environment like VSCode into a production-ready app using Reflex, applying both frontend and backend techniques. We also learned to prompt engineer LLM models like Gemini for our specific use case. We also learned how to implement Natural Language Processing (NLP) techniques to extract meaningful insights from the call logs to show user progress.

What's next for Webly

In the future, we would like to develop Webly into a Chrome extension that automatically understands whether the user completed the task or not rather than an external app platform. We would also like to enable the user to set up how the voice interface sounds as they prefer. Overall, we hope to grow Webly into a quick and easy-to-use platform which can help older adults navigate the Internet for whatever they need.

Built With

• css
• gemini
• html
• hume.ai
• python
• reflex