Inspiration
What it does
How we built it
Challenges we ran into## About the Project
Inspiration
Voice AI has the potential to transform customer service and lead qualification, but experimenting with voice-calling workflows can involve additional costs and infrastructure for telephony services. I wanted to build a practical way to develop and test an AI calling agent without depending on paid phone integrations during development.
This led me to build Local AI Voice Calling Agent Simulator — an interactive application focused on real estate lead qualification and automated customer service.
What It Does
The application simulates incoming calls using uploaded audio files. It transcribes the caller's speech, identifies their intent, extracts useful details such as name, budget, location, and timeline, and classifies prospects as HOT, WARM, or COLD.
Based on the conversation context, the AI generates a relevant voice response. A real-time interface displays transcripts and analysis, while a CRM-style lead board and analytics dashboard help organize and review prospects.
The goal is to go beyond simply generating an AI response and demonstrate how voice interactions can feed into a structured business workflow.
How I Built It
I built the application with a React 19 and Vite frontend and a NestJS backend. The system combines several components into a single processing pipeline:
- Speech-to-Text: OpenAI Whisper transcribes uploaded audio.
- AI Analysis: An LLM accessed through OpenRouter analyzes intent, extracts lead information, and helps generate contextual responses.
- Text-to-Speech: ElevenLabs or Google TTS converts generated responses into audio.
- Real-Time Updates: Socket.IO communicates transcript and analysis updates between the backend and frontend.
- State and History: Redis supports caching, call state, and conversation history.
The frontend brings these capabilities together in a dashboard with a call simulator, live transcript, lead management board, and analytics.
Challenges I Faced
One challenge was building a multi-stage voice workflow where transcription, AI analysis, and voice generation work together. Each stage depends on the output of the previous one, so the application needs to manage call state and communicate updates clearly.
Another challenge was working with external AI services while developing within practical API limits. I incorporated LLM provider rotation through OpenRouter and fallback options for speech generation and transcription-related processing to make development and testing more resilient.
Keeping the interface synchronized with backend processing was also important. Socket.IO helps deliver call transcripts, analysis updates, and generated audio information to the frontend in real time.
What I Learned
Building this project strengthened my understanding of integrating speech recognition, LLM orchestration, text-to-speech, backend APIs, real-time communication, and structured lead management into one application.
I also learned that a useful AI agent is about more than conversation. It should extract actionable information, maintain context, and connect its outputs to a workflow that users can actually work with.
What's Next
The current project focuses on local call simulation using audio files rather than live telephone integration. The next steps are to improve the conversation workflow, strengthen reliability, and explore integrations that make the agent useful in more realistic voice-first customer service scenarios.
Accomplishments that we're proud of
What we learned
What's next for AI Voice Calling Agent Simulator
Built With
- llm
- nest
- reactt
Log in or sign up for Devpost to join the conversation.