Vera.ai

The project logo.
Knowledge Base Worflow.
Agent Creation Workflow.-1
Agent Creation Workflow.-2
Database Schema.

vera.ai — Project Story

Inspiration

Real-world friction: During an internship developing voice agents for dental clinics, I tested existing platforms and encountered significant roadblocks: high latency, prohibitive costs, and rigid setups that failed to meet clinical needs.

The "Wrapper Tax" is killing voice AI. While building agents for enterprise clients, we realized the current market leaders (Vapi, Bland, Retell) rely on "Cascade Architectures": stitching together Deepgram (transcription), GPT-4 (reasoning), and ElevenLabs (speech). This introduces massive latency (~1.5s), strips away emotional nuance, and inflates costs to ~$0.10/min. We saw the release of Gemini’s Native Audio capabilities as a paradigm shift. We wanted to build the first "Native Infrastructure" platform—stripping away the middlemen to connect developers directly to the edge for <200ms latency and 99% lower costs.

What it does

Vera.ai is a Prompt-to-Infrastructure platform for the next generation of voice agents. Orchestrator Engine: You describe an agent (e.g., "A HIPAA-compliant dental scheduler"), and a squad of Gemini 3 agents architect the solution, auto-generating tools and compliance guardrails. Native Audio Core: A runtime environment that bypasses STT/TTS entirely, streaming raw PCM audio directly to gemini-2.5-flash-native-audio. Neural Squad Builder: A visual node editor where users define sub-agents. A "Root Router" (Gemini 3 Flash) analyzes conversation intent in real-time to route calls between specialists (e.g., passing a user from "Reception" to "Billing" seamlessly). Deep Grounding: An integrated RAG system where Gemini scours the web to build a "Knowledge Brain" for the agent instantly.

How we built it

Frontend: React 19, TypeScript, and TailwindCSS for a futuristic, fluid UI. Audio Engine: We bypassed standard browser media recorders. We implemented a custom AudioWorklet to capture raw Float32 microphone data, downsample it to 16kHz PCM16, and stream it via WebSockets using the @google/genai SDK. Intelligence Layer: Orchestration: gemini-3-pro-preview analyzes user prompts to generate JSON blueprints. Streaming: gemini-3-pro-native-audio-preview handles the bi-directional audio stream. Routing: gemini-3-flash-preview is used in the Squad Builder to detect "Intent Collisions" between agents. Visuals: Dynamic SVG rendering for the Squad Builder and Canvas-based audio visualizations (Orb) that react to real-time volume amplitude.

Challenges we ran into

Raw Audio Processing: Browsers don't natively output the specific PCM16 little-endian format Gemini requires. Writing the binary conversion logic in the browser while maintaining zero latency was a significant hurdle. Barge-In Logic: Handling interruptions naturally is difficult. We had to implement precise state management to flush audio buffers the millisecond the user starts speaking, ensuring the agent stops talking immediately—just like a human. Neural Routing: Visualizing non-deterministic AI routing was complex. We built a simulation engine that allows developers to type a phrase and "watch" Gemini weigh the probabilities of routing to different nodes in the visualizer.

Accomplishments that we're proud of

<200ms Latency: We proved that by removing the "wrapper stack" (Deepgram/ElevenLabs), we could achieve human-level response times. Cost Efficiency: We drove costs down from the industry standard of ~ 0.0006/min** by using direct token inference. The Squad Builder: Creating a drag-and-drop interface that isn't just a flowchart, but a live representation of a Neural Network's decision-making process.

What we learned

We learned that Native Audio is not just speed; it's EQ. Traditional text-based LLMs lose prosody, sarcasm, and hesitation. By feeding raw audio tokens to Gemini, Vera understands how a user is speaking (angry, confused, rushing), not just what they are saying. This unlocks a level of empathy previously impossible in Voice AI.

What's next for vera.ai

Vera Vision: Fully implementing the video stream capabilities to allow agents to "see" via the user's camera during support calls. One-Click Telephony: Automating the SIP trunking process to deploy these native agents to phone numbers globally instantly. MCP Marketplace: Expanding our tool definitions to support the Model Context Protocol for instant integrations with Linear, Salesforce, and Notion.