Hermes

• 

  Agents

• 

  Call Breakdown

• 

  Dashboard

• 

  Landing

💡 Inspiration

As immigrants, we grew up making calls for our parents who struggled with English. That meant sitting through 40+ minute hold times with immigration offices, hospitals, and more. We built Hermes to be a smart AI could wait, talk, and act on their behalf.

✉ What it does

Hermes is an AI calling agent. You give it a prompt, and it:

• Calls the number and waits on your behalf.
• Navigates menus & talks naturally with humans or IVRs.
• Transfers call to actual user if it doesn't have sufficient info
• Controls integrations like Calendar or Maps.
• Reports back with a recording and summary to answer any questions.

💻 How we built it

We built Hermes on top of the Vapi voice + telephony stack, using large language models to power natural conversation. Our pipeline connects:

• Vapi for real-time calling and speech synthesis.
• ElevenLabs and Deepgram for TTS and STT
• Gemini + OpenAI for dialogue understanding and reasoning.
• Custom tool integrations (Google Calendar in our demo) to let Hermes act beyond the call.
• A dashboard UI to manage calls, view transcripts, and track results.

⚠️ Challenges we ran into

Outbound limits – Free mobile numbers for outbound calls ran out, so we had to switch to Vonage and optimize quickly.

STT breakdowns – Deepgram crashed before submission, forcing us to experiment with different models and audio channels under time pressure.

Vapi docs maze – The documentation was complex and unclear, so just getting Hermes stable took serious digging and trial-and-error.

🎻 What we learned

How to stitch together real-time voice, AI reasoning, and tool control into one seamless pipeline.

The importance of fallbacks, when one service failed (like STT), we had to pivot fast.

That UX matters even in backend-heavy projects: transcripts, summaries, and clean flows made Hermes feel usable.

Most of all, we learned that building an AI phone agent isn’t just coding, it’s solving messy edge cases in the real world.

Built With

• browser-use
• chromium
• deepgram
• elevenlabs
• fastapi
• nextjs
• python
• shadcn
• supabase
• tailwind
• typescript
• vapi