Video Conferencing with Conversational AI

• 

  home

Inspiration

Remote meetings have become the backbone of modern work, but they're broken. We've all been in those moments where someone asks "What did we decide about the budget?" and everyone scrambles through old notes. Important decisions get lost, new team members miss crucial context, and meetings feel like endless repetition. We wanted to build an AI that doesn't just sit silently in the corner taking notes, but actually participates as a knowledgeable team member who remembers everything.

What it does

Our AI Meeting Platform creates video conferences where an intelligent voice assistant joins as a participant with perfect institutional memory. The AI can instantly search through all past meeting knowledge, recall specific decisions, summarize topics, and provide contextual insights during live conversations. Users can ask questions like "What were our Q3 goals?" or "Show me all action items from last month" and get immediate, accurate responses. The platform combines real-time video conferencing with a RAG-powered knowledge base that builds organizational memory across all meetings.

How we built it

We built a full-stack application using React and TypeScript for the frontend, with WebRTC handling peer-to-peer video connections through a Node.js signaling server. The AI integration uses ElevenLabs for conversational voice capabilities and OpenAI for generating embeddings that power semantic search. We implemented a vector database using Supabase with pgvector extension to store and search meeting knowledge. The backend includes comprehensive API endpoints for knowledge management, secure authentication through Clerk, and real-time synchronization. We also built extensive testing infrastructure and configured CORS for multi-environment deployment including Netlify and cloud IDEs.

Challenges we ran into

WebRTC connection management proved complex, especially handling peer-to-peer connections across different network configurations. Integrating real-time voice AI while maintaining low latency was challenging. We spent significant time optimizing the RAG system to provide relevant search results and prevent hallucinations. Database schema design for storing embeddings and ensuring efficient vector similarity search required multiple iterations. Authentication and security presented ongoing challenges, particularly implementing proper access controls without breaking the user experience. Finally, coordinating real-time updates between the AI knowledge base and live meeting participants required careful state management.

Accomplishments that we're proud of

We successfully created a working video conferencing platform with integrated conversational AI that actually understands meeting context. The RAG system delivers accurate, relevant results from meeting history. Our testing suite covers comprehensive API functionality and integration scenarios. We implemented proper production-ready security measures and deployment configurations. The AI tools integration with ElevenLabs works seamlessly, allowing natural voice interactions during meetings. We built a scalable architecture that can handle multiple concurrent meetings and knowledge bases. The user experience feels natural and intuitive, making AI assistance feel like having a super-smart colleague who never forgets anything.

What we learned

Building real-time applications requires careful consideration of state management and network reliability. AI integration is about more than just API calls; it requires thoughtful prompt engineering and context management. Vector databases and semantic search are powerful but need proper optimization for production use. Security cannot be an afterthought, especially when dealing with sensitive meeting data. Testing infrastructure is crucial for complex integrations involving multiple external services. User experience design for AI features requires balancing capability with simplicity. Performance optimization becomes critical when combining real-time video, voice AI, and database operations.

What's next for Video Conferencing with Conversational AI

My first priority is adding web search with Perplexity. We plan to add advanced features like automatic meeting summaries, intelligent action item tracking, and proactive insights based on meeting patterns. Integration with popular productivity tools like Slack, Notion, and project management platforms will make the AI a central hub for team knowledge. We want to implement advanced search capabilities including visual content analysis and document integration. Multi-language support and enterprise-grade security features are priorities for broader adoption. Long-term, we envision the AI becoming a strategic advisor that can analyze meeting patterns, suggest process improvements, and help teams make better decisions based on historical data and trends.

Built With

• elevenlabs
• express.js
• node.js
• react
• supabase