APEX CONNECT AI

An AI Voice agent that executes tasks and books meetings seamlessly through natural voice commands. (Can be used in a corporate environment for seamless meeting booking with clients or to get data)

Comment

Inspiration

Between juggling daily tasks, follow-ups, and calendar management, booking appointments and qualifying leads manually takes up far too much time. I wanted to build an intelligent, hands-free assistant that handles the entire interaction end-to-end - acting like a personal phone assistant that contacts leads, asks when they'd like to schedule, captures their details, and handles all the calendar and email setup automatically.

What it does

It acts as your personal AI phone assistant and schedule coordinator. It reaches out to leads or handles incoming requests over voice to collect their scheduling preferences and gather essential details (including their email address). Once confirmed, It automatically books the slot directly into your Google Calendar and instantly sends a confirmation email with all the details to the lead.

How we built it

I implemented low-latency Speech-to-Text (STT) and Text-to-Speech (TTS) pipelines over WebSockets to maintain natural, fluid turn-taking during conversations. The frontend is built with pure HTML/JS using the Web Audio API for a custom visualizer.

For the brain, I built a Python Flask backend. The voice interaction is handled via Gemini's multimodal API, but I integrated the Featherless API for the backend intelligence. I used Featherless to handle dynamic text generation and intent parsing in the background. Finally, I used Google Cloud's OAuth 2.0 to securely bridge the app directly to the native Google Calendar and Gmail APIs.

Challenges we ran into

Handling the asynchronous nature of voice conversations was definitely the hardest part. People pause, say "um," or change their minds mid-sentence. Getting the tool-calling right so the AI actually stops talking, queries my Flask backend for calendar availability, waits for the response, and then resumes the conversation naturally took a lot of trial and error and console logging.

Also, time zones! Parsing ISO timestamps and ensuring the calendar didn't accidentally book a lead at 3 AM local time was a huge headache that required writing some strict datetime conversion rules in Python.

Accomplishments that we're proud of

I'm incredibly proud of the latency. The system genuinely feels like you're talking to a real human on the phone. Achieving that sub-second response time while simultaneously making live API calls to Google Calendar and Featherless API in the background is a huge win. I'm also really proud that I built the entire audio visualizer from scratch using raw audio buffers rather than relying on heavy third-party UI plugins.

What we learned

I learned a ton about WebSockets and streaming raw audio data between a browser and a server. It requires a completely different mindset compared to standard REST APIs. I also got really comfortable managing OAuth tokens and navigating the quirks of the Google Workspace ecosystem. Integrating the Featherless API taught me how incredibly easy it is to plug powerful open-source models into a backend pipeline when you need scalable, serverless intelligence.

What's next for APEX CONNECT AI

The immediate next step is Twilio integration, so this isn't just a web app, but an actual phone number that clients can dial directly from their cell phones. I plan to build out a dashboard where users can view call transcripts, track lead analytics, and set custom routing rules (like "never book meetings on Fridays") and can also use it for taking surveys of people.

Built With

Share this project:

Updates

Submission history