Visionary Agent

Meet your new compliance partner: a fully animated, voice-driven AI auditor powered by the Google Agent Development Kit.

Comment

Inspiration

Enterprise compliance, auditing, and organizational policies are historically dry and frustrating to navigate. Traditional text-based chatbots often feel like glorified search bars, they lack empathy, personality, and the ability to truly collaborate. For the "Collaborative Partner" track, we wanted to build an AI that feels like a real teammate. We were inspired to bridge the gap between cold data and human interaction by creating an embodied, face-to-face virtual auditor that not only knows the rules but adapts to how you uniquely prefer to work.

What it does

Visionary Agent is an embodied 3D virtual auditor that guides users through complex compliance rules. Instead of reading blocks of text, users interact with a fully animated, voice-driven 3D avatar.

  • It Knows: It uses RAG to pull grounded facts directly from organizational PDFs.
  • It Learns: It actively listens for your workflow preferences (e.g., "give me bullet points," "skip the intro") and saves them to a long-term memory database, isolating those preferences to your specific session.
  • It Reacts: The 3D avatar features procedural breathing, darting eye saccades, and real-time lip-syncing driven directly by the agent's synthesized voice.

How we built it

We engineered a decoupled, real-time architecture utilizing the Google Cloud ecosystem:

  • The Brain (Backend): We used FastAPI and the Google Agent Development Kit (ADK) orchestrated by Gemini 3. The ADK is equipped with custom Python tools that query a Firestore Vector Database for document context, and read/write to a user_preferences collection to maintain state.
  • The Voice: We stream responses through Google Cloud Text-to-Speech, utilizing an async worker thread to chunk text and generate base64-encoded MP3 audio on the fly.
  • The Face (Frontend): We built an Angular application rendering a facecap.glb model using Three.js. We established a WebSocket connection to stream the audio and text seamlessly.
  • The Animation: Using the browser's native Web Audio API, we analyze the frequency of the incoming audio stream frame-by-frame and map the volume amplitude directly to the avatar's ARKit jawOpen blendshape to achieve zero-latency lip-syncing.

Challenges we ran into

  • TTS Hallucinations from Markdown: LLMs naturally output markdown (asterisks, hashes, bolding), which completely breaks Text-to-Speech engines, making the agent speak out punctuation marks awkwardly. We had to build a custom regex sanitization pipeline inside the backend to clean the ADK's output before synthesizing the voice.
  • User Data Isolation over WebSockets: We wanted the agent to remember user preferences without forcing judges through a complex login flow. We solved this by generating an anonymous guest_id in the browser's localStorage and attaching it to every WebSocket payload. On the backend, we used Python's ContextVars to ensure the ADK tools always read/wrote to the correct user's Firestore document asynchronously.
  • Static "Ventriloquist Dummy" Effect: Initially, just moving the jaw made the 3D model look lifeless. We had to dive deep into Three.js math to program continuous sine-wave head swaying and randomized blendshape eye saccades to introduce procedural human "noise" to the mesh.

Accomplishments that we're proud of

We are incredibly proud of successfully transitioning from a standard prompt-response chatbot into a true Agentic workflow. By utilizing the Google ADK to orchestrate tools, the AI actively decides when to search the knowledge base and when to log a new user preference on its own. Pairing that advanced backend reasoning with a highly responsive, custom-built WebGL frontend resulted in an MVP that feels genuinely alive and production-ready.

What we learned

  • The profound difference between standard LLM calls and true agentic frameworks like the Google ADK. Giving an agent its own tools changes how you architect an entire application.
  • Advanced Three.js optimization techniques, including using KTX2Loader and MeshoptDecoder to compress 3D assets so the frontend loads instantly.
  • How to effectively handle streaming data over WebSockets using exponential backoff to handle transient API limits gracefully.

What's next for Visionary Agent

Our immediate next step is expanding the multimodal capabilities. We plan to integrate the user's webcam and screen-sharing, allowing the Visionary Agent to "see" the documents a user is actively auditing on their desk and offer real-time, vocal corrections. We also want to implement a seamless interruption feature, allowing users to cut the agent off mid-sentence with their microphone if they need immediate clarification.

Built With

Share this project:

Updates