VoNav

Inspiration

Screen readers require learning a lot of hotkeys and aren't very user friendly. They also don't just "click" stuff for you - you have to learn it like a new language.

What it does

Uses your voice to control the computer and read what's on it.

How we built it

Livekit with custom Playwright tools, with a debug mode chrome instance running.

Challenges we ran into

1. Tried using Playwright mcp, but it didn't work with the browser cookies.
2. Tried using Agno mcp, but couldn't get it to work.
3. The tools weren't being activated, which was an issue of google's api, so I switched to open ai
4. The realtime model kept saying it couldn't do something, but this was fixed with prompt fixing. ## Accomplishments that we're proud of Getting the AI to launch YouTube, and play a video! ## What we learned How to make a voice agent on Livekit that uses tools, and how to use Playwright. ## What's next for VoNav Package it into an app that anyone can download, and make the voice agent speak faster. Video for demo: https://www.loom.com/share/939c6fcd5da441f49f99bc414cd2fba2?sid=c9b7ae89-7a3f-417c-aa8d-8c28d551616f

Built With

• chrome
• livekit
• playwright
• python