• 

  Architecture

Inspiration

We noticed that people preparing for high-stakes conversations: investor pitches, enterprise sales calls, job interviews, onboarding walkthroughs, had no way to practice with a realistic counterpart that could actually see their slides, hear their voice, and push back intelligently. Existing tools were either text-based chatbots or rigid role-play scripts. We wanted to build something where you could create any AI persona, feed it real context from your emails and documents, and have a live, multimodal conversation that feels real.

What it does

Falah.ai lets users build custom AI agents with configurable personalities, voices, focus areas, and scoring criteria, then go live with them using real-time audio and screen sharing. The AI agent can see the user's screen (slides, dashboards, documents) and respond to what's visible. After every session, Gemini scores performance across multiple dimensions and generates actionable feedback. Users can upload context documents (emails, call transcripts, briefs) that get synthesized into an intelligence dossier the agent uses naturally during conversation. The platform also adapts across sessions, probing weak areas from previous attempts and escalating difficulty.

How we built it

The backend runs two independent processes: a FastAPI REST server for authentication, agent management, and session tracking, and a LiveKit Agent Worker that handles real-time AI conversations. When a user starts a session, the server pre-bakes all persona configuration, context briefings, and session history into LiveKit room metadata, so the agent worker makes zero database calls during live conversations, keeping latency minimal.

The agent worker connects to Gemini 2.5 Flash's native audio and vision model through LiveKit's Agents SDK. Audio streams bidirectionally in real-time, while screen share frames are adaptively sampled using Voice Activity Detection, 0.5 FPS while the user speaks (capturing important slides) and 0.2 FPS during silence (conserving tokens). We configured Gemini with a thinking budget of just 128 tokens for near-instant responses, affective dialog to adapt tone to the user's voice, and context window compression (triggering at 90K tokens, compressing to 45K) for long sessions.

Context intelligence uses Gemini Embedding 2 to generate 3072-dimensional vectors stored in PostgreSQL with pgvector, enabling semantic search across uploaded documents. Briefings are synthesized by Gemini Flash into structured dossiers (key concerns, decision factors, data points) and cached on the persona record.

Post-session scoring uses Gemini Flash to evaluate transcripts against custom or default dimensions (1-10 each), generating both scores and written feedback. The frontend is built with React 19, Tailwind CSS, and LiveKit's React SDK for WebRTC integration.

Challenges we faced

Event loop isolation: The LiveKit agent worker and FastAPI server each run their own async event loop. Sharing a single SQLAlchemy engine caused "Future attached to a different loop" errors. We solved this by spawning a fresh, isolated database engine scoped to each agent worker invocation. Token budget management: With audio, vision, persona config, context briefings, and session history all competing for Gemini's 128K context window, we had to carefully cap each source (CRM: 1K chars, briefing: 2K chars, session history: 1 most recent) and implement sliding window compression. Vision frame rate tuning: Too many screen frames bloated the context and slowed responses. Too few made the agent miss important slides. Voice Activity Detection-based adaptive sampling was the sweet spot. Latency vs intelligence: Setting the thinking budget to 128 tokens gave us conversational-speed responses (~100-300ms reasoning) but required well-crafted system prompts to compensate for reduced reasoning depth.

Accomplishments that we're proud of We built a fully functional real-time multimodal AI platform from scratch — not a demo, but a product people can actually sign up for and use. The moment we got an AI agent to see a user's screen, hear their voice, and respond naturally in real-time with under 300ms reasoning latency was a breakthrough moment. Specifically:

Zero-DB-call live sessions: By pre-baking all persona config, context briefings, and session history into LiveKit room metadata at token time, the agent worker makes no database calls during a live conversation keeping the experience instant and fluid. Adaptive vision: Our Voice Activity Detection-based video sampling (0.5 FPS during speech, 0.2 FPS during silence) solved the tension between the AI seeing enough of the user's screen and not blowing up the context window. A truly general-purpose platform: We resisted the temptation to hard-code it as a pitch simulator. Any conversation skill :sales, interviews, onboarding, training, knowledge transfer, support works because users define the agent's personality, focus, and scoring criteria themselves. The 7 starter templates prove the range. Voice-driven onboarding: New users are greeted by a live AI agent ("Falah") that can see their screen and walk them through the platform conversationally, no tutorials, no docs, just a real-time voice guide. End-to-end session intelligence: Upload context docs, get AI-synthesized briefings, go live with an adaptive agent, then receive scored feedback with actionable dimension-level insights, all in one flow.

What we learned Gemini's native audio + vision model is production-ready for real-time use cases, but tuning the thinking budget is critical. 128 tokens gives conversational speed; anything higher introduces noticeable pauses that break immersion. Context window management is an engineering discipline, not an afterthought. With audio, video frames, persona prompts, briefings, and session history all competing for 128K tokens, we learned to budget aggressively: cap every source, compress proactively at 90K, and sample video adaptively. Two-process architecture is non-negotiable when combining FastAPI with LiveKit agents, their async event loops conflict if you try to share a database engine. Spawning isolated engines per worker was the clean fix. Prompt engineering scales with structure: Our _build_dynamic_prompt() function taught us that well-sectioned prompts (identity, personality, dossier, history, rules) perform dramatically better than flat text, especially when the model needs to juggle multiple context sources. Post-call scoring with a cheaper model works: Using Gemini Flash for scoring instead of the live model saves cost and doesn't block the user experience. The neutral fallback scorecard (all 5.0 on parse failure) keeps the UX graceful even when scoring fails.

What's next for Falah.ai Connectors: Gmail, Google Docs, Slack, Salesforce, HubSpot, and Notion integrations are scaffolded in the UI, next step is wiring them up so agents automatically pull context from real-world sources before sessions. Session resumption: Gemini supports session handles for reconnection. We've configured the infrastructure but haven't implemented persistence yet, this would let users resume interrupted sessions seamlessly. Real-time streaming scores: Currently scoring happens post-session. We want to explore live, incremental scoring so users get feedback during the conversation, not just after. Multi-language support: All prompts are English-only today. Gemini's multilingual capabilities would let us serve users globally. Team workspaces: Enterprise teams could share agent libraries, track team-wide scoring trends, and run structured training programs with assigned agents and progression paths. Agent-to-agent conversations: Let two AI agents practice against each other to stress-test persona configurations before a human goes live, essentially an AI sparring mode.

Built With

• alembic
• docker
• embedding-2
• fastapi
• gemini
• gemini-2.5-flash-(native-audio-+-vision
• google-genai
• google-genai-sdk
• languages:-python
• livekit
• pgvector
• postgresql
• react
• sqlalchemy
• tailwind
• typescript
• uvicorn