Inspiration
Like Tony Stark's J.A.R.V.I.S., I wanted an assistant that's more than a talking search box: one with a personality, that you can just tell what to do, and that does it. Today's voice assistants talk, but they can't actually use your computer. So I built Vesper, a voice assistant that lives in your browser, hears you, and does the clicking for you, with a mood you can hear and safety you can trust.
What it does
Vesper is a voice assistant that lives in your browser. You talk to it, and it controls your tabs with its own visible cursor: it opens and searches YouTube, Google, Maps and Gmail, scrolls, plays, pauses and seeks videos, clicks results, switches tabs, and reads what is on your screen back to you. It also works on GitHub, Reddit, Notion, LinkedIn and Discord tabs, and opens Spotify, X and ChatGPT.
It has a real mood. Vesper stays warm by default, gets annoyed or angry if you push it, and cools down gradually over the conversation, and it still does the task while it grumbles. The voice changes with the mood.
Vesper is built to be careful: it asks you out loud before anything that sends, posts, buys or deletes, says exactly what will be sent and where, and refuses to touch password, payment or verification fields. Text on a web page is treated as data, never as instructions.
How we built it
- Voice: ElevenLabs Agents with expressive mode (eleven_v3_conversational), custom audio tags for emotion, and client tools for every action. A Claude Haiku model drives the conversation.
- Browser control: a Chrome extension (Manifest V3) takes a numbered snapshot of the visible page, a Gemini model chooses the next action, and the extension performs it with a visible "Vesper" cursor using real, trusted input events through the Chrome debugger API.
- Honest verification: every action checks its own effect (did the URL change? did the text appear?) and reports failure plainly instead of claiming success.
- App: Next.js, TypeScript and Tailwind, with Supabase for the memory schema.
Challenges we ran into
Websites ignore scripted clicks, so we moved to trusted input events. Our own tools used to say "done" when nothing had happened, which made the agent confidently lie, so we rebuilt them to verify results. Rich-text editors like Discord's message box ignore normal typing, so we send real key events. Getting an emotion to persist and cool down gradually, instead of snapping back, took careful prompt design.
Accomplishments that we're proud of
An assistant that talks with real emotion and can actually operate a browser, with a visible cursor and safety rules enforced in code, not just in the prompt.
What we learned
Agents need verified tool results, not optimistic ones. An assistant that says
Built With
- chrome
- claude
- elevenlabs
- gemini
- next.js
- react
- supabase
- tailwindcss
- typescript
