Inspiration

AI is great at planning but stops short of doing. Ask an assistant to book a table at a restaurant and you get a phone number back, and you still have to make the call yourself. We wanted an agent that picks up the phone and gets it done.

What it does

You create an AI agent and choose what it's allowed to do: phone calls, email, calendar, payments. Then you just ask: "Book a dentist appointment tomorrow at 3pm, 2 or 2:30 also works." Smart Connect plans the call, a voice agent phones the business and handles the conversation, and you get the transcript, a summary and the confirmed booking, plus a calendar entry and confirmation email. If your time isn't free, it only offers the alternatives you allowed, and it never pays a deposit without your approval.

How we built it

Gemini plans the call with function calling, then reads the transcript afterwards to work out the result and decide what to do next.
ElevenLabs + Twilio make the real phone call and send back the transcript through a signed webhook.
Tiger Data stores each call as a time-series of events in a TimescaleDB hypertable, which powers the live call view.
Vultr hosts the FastAPI backend and React frontend.

Challenges we ran into

The hardest part was connecting all the different tools together and getting them to work.

Accomplishments that we're proud of

It makes real phone calls end to end. You can type a message, a phone rings, the AI agent speaks on your behalf, and the booking shows up on your calendar :)

What we learned

Almost everything in this project was new to us. We'd never built a voice agent, connected an AI to a real phone line, or used a time-series database. Along the way we learned how to:

Get an AI to use tools with Gemini function calling, and pause it mid-task while a call happens.
Build and tune a voice agent with ElevenLabs, and connect it to real phone calls through Twilio.
Receive and verify webhooks, so other services can securely tell our app when something happens.
Store a phone call as a time-series of events in TimescaleDB, and stream it to the browser live.
Deploy a real app on our own Vultr server, with HTTPS, and debug it from its logs.