Waypoint

Waypoint - talk to Clara, Kingsford University's real-time voice AI counsellor, powered by Google ADK and Gemini Live.

Comment 1

Inspiration

University course selection is overwhelming. Prospective students — especially international students navigating visa requirements, ATAR cutoffs, and scholarship deadlines from overseas — face hundreds of options and scattered information across brochures, PDFs, and university websites. Open days are busy. Admissions staff are stretched. And students often give up before they find the answer they need.

We wanted to see what happens when you put a knowledgeable, always-available voice AI counsellor in front of them — one that listens, understands their situation, and responds instantly with grounded, accurate information. Not a chatbot. A conversation.

We also wanted to push Google ADK beyond its comfort zone: into real-time bidirectional audio with tool use, vision input, and a live card UI — and see how far we could get in ten days.

What it does

Waypoint lets prospective students talk to Clara, Kingsford University's AI course counsellor, in real time. She hears you, sees what you share, retrieves grounded university data, and responds with audio and structured visual cards — simultaneously.

Voice-first conversation: Speak naturally, interrupt mid-sentence (barge-in), and Clara responds with audio instantly. No typing required.

Grounded answers only: Clara never invents a course name, fee, or ATAR score. Every factual response comes from a tool call against a live database. We ran a 3-layer automated eval suite — 63 out of 64 test cases passed (98%).

Real-time research cards: As Clara speaks, structured cards appear in a sidebar without waiting for her to finish. Course tiles, scholarship cards, markdown knowledge cards (with rendered tables), and booking confirmations all arrive via a WebSocket side-channel.

Vision input: Students can share a photo — a school transcript, a merit award, or a Google Maps view of campus — and Clara responds in context. Share a transcript → she reads the grades and recommends matching courses. Share a campus map → she retrieves grounded information about that building from the knowledge base.

Full agentic actions: Clara doesn't just answer questions — she books campus tours, writing confirmed reservations to a live Cloud SQL database with a GT-XXXXX reference number.

Accessibility toggle: One click switches Clara into Text Mode, silencing both the microphone and her voice for a seamless text-only experience in quiet environments.

8 tools, all firing: | Tool | What it does | |------|-------------| | search_courses | pgvector semantic search — "I like solving problems" finds Engineering | | get_course_detail | Full detail card for a named course — fees, ATAR, career outcomes | | recommend_courses | Personalised recommendations from interests, strengths, study mode | | search_events | Upcoming open days, info sessions, campus tours | | search_knowledge | RAG over 50 knowledge chunks — ATAR, HECS-HELP, visa, facilities | | search_scholarships | 8 scholarships with type filters (Merit, Equity, International, Faculty) | | book_campus_tour | DB insert → GT-XXXXX booking reference | | display_data | Side-channel → WebSocket → frontend cards (fires silently alongside speech) |

How we built it

The core is a FastAPI WebSocket bridge connecting a browser microphone to Gemini 2.5 Flash Native Audio via Google ADK's Runner.run_live() API.

Audio pipeline: Browser AudioWorklet captures 16kHz PCM → WebSocket → ADK LiveRequestQueue → Gemini Live BIDI stream → PCM audio back → browser AudioContext plays in real time.

Vision pipeline: Browser captures image frames as JPEG → base64 encoded → sent as Blob(mime_type="image/jpeg") via LiveRequestQueue alongside audio → Gemini processes multimodally.

Tool orchestration: ADK routes function_call events from Gemini to the correct Python tool. Tools query Cloud SQL via psycopg2 + pgvector (cosine similarity with 1536-dim gemini-embedding-001 embeddings). Tool results are returned to Gemini and also pushed directly to the browser via a display_data side-channel, so cards appear before Clara finishes speaking.

Card side-channel: display_data is an ADK tool that, instead of returning data to the LLM for summarisation, serialises a card JSON payload and sends it directly over the WebSocket to the browser — rendering course tiles, scholarship cards, and markdown info cards in real time.

Frontend: Single-file HTML/JS with no build step. AudioWorklet mic capture, real-time delta-based transcript streaming, wave animation synced to both user and agent speech, draggable panel resize slider, card group history with icon pills.

Database: Cloud SQL PostgreSQL 16 + pgvector. Tables: courses (16), events (10), scholarships (8), knowledge_docs (50 chunks across 6 markdown files), tour_bookings. HNSW cosine indexes on embedding columns.

Deployment: Cloud Run (us-central1, min-instances=1 to eliminate cold starts, 1Gi memory). Cloud Build CI/CD pipeline (cloudbuild.yaml). Database credentials via Secret Manager.

Evaluation: 3-layer automated eval suite (eval_suite.py):

  • Layer 1 (23 assertions): Direct DB tool calls — all 8 tools return correct, well-shaped responses
  • Layer 2 (24 routing queries): Natural language → Gemini text API → correct tool selected
  • Layer 2b (17 multi-turn turns): 4 full counselling conversations with context-aware routing

Result: 63/64 passed (98%).

Challenges we ran into

ADK audio wire format: ADK's GeminiLlmConnection.send_realtime() uses a deprecated mediaChunks wire format. The native audio model requires the audio key. We monkey-patched the connection class to fix this — without it, the model's voice activity detection doesn't trigger.

Tool response camelCase conversion: ADK's send_tool_response() applies recursive camelCase conversion to tool result keys (e.g. career_outcomes → careerOutcomes). The native audio model rejects this with a 1011 internal error. We bypassed send_tool_response() entirely, constructing and sending the function response JSON manually over the raw WebSocket connection.

context_window_compression not supported: ADK's SlidingWindow compression is not supported in bidirectional audio mode. Attempting to enable it causes 1008 disconnects. Removed entirely from RunConfig.

Session stability: The Gemini Live connection can return 1007 ("Precondition check failed") if the ADK Runner or SessionService is created at module scope and reused across connections. Fix: create a fresh Runner per WebSocket connection, with a module-scoped InMemorySessionService to preserve conversation history across reconnects.

Transcript deduplication: ADK sends both incremental partial transcripts and a final complete transcript. Without careful delta-tracking, text flashes and duplicates in the UI. We implemented delta-based streaming — partials extend the display incrementally, and the final transcript replaces only the accumulated partial run.

TURN ISOLATION: The native audio model produces garbled or empty audio if a tool call and spoken text occur in the same turn. Clara's instruction strictly enforces: call a tool silently, then summarise in the following turn.

Accomplishments we're proud of

  • Card side-channel: Visual cards arrive in the browser while Clara is still speaking — not after. This required a custom side-channel that bypasses the normal LLM response loop entirely.
  • Vision + grounding combo: Clara sees a Google Maps photo of the campus and retrieves grounded information about the building from the knowledge base — not a hallucination, a real pgvector lookup.
  • 98% eval accuracy: 63/64 automated test cases passing, with documented methodology and reproducible results in the public repo.
  • Zero hallucinations in testing: In every observed session, Clara retrieved data from tools before stating any fact.
  • Production-grade deployment in 10 days: Cloud Run + Cloud SQL + Secret Manager + Cloud Build CI/CD, all from scratch.

What we learned

Google ADK is a powerful orchestration layer, but real-time native audio is still frontier territory. The gap between "text agent that works" and "voice agent that feels natural" is significant — audio wire format, VAD behaviour, turn isolation, transcript timing, and connection stability all require careful engineering that isn't yet documented.

The display_data side-channel pattern was one of the most valuable insights: separating the visual output channel from the spoken output channel makes multimodal UX feel coherent, not cluttered.

pgvector semantic search with gemini-embedding-001 is remarkably good out of the box. "I love maths and want a stable career" reliably surfaces Engineering and Data Science programs with no prompt engineering on the search side.

What's next for Waypoint

  • Multi-university support: Clara's architecture is institution-agnostic — the knowledge base, tools, and card UI can be reseeded for any university with minimal changes
  • Persistent student profiles: Replace InMemorySessionService with database-backed sessions so Clara remembers a student's preferences across visits
  • Proactive nudges: Push scholarship deadline reminders and open day alerts via email/SMS — moving Clara from reactive Q&A to proactive guidance
  • Richer vision flows: Scan a fee invoice, timetable, or offer letter and have Clara explain it in context
  • Mobile app: The WebSocket + AudioWorklet stack maps cleanly to a React Native wrapper for iOS/Android

Built With

python · javascript · google-adk · gemini-2.5-flash-native-audio · vertex-ai ·
fastapi · uvicorn · websocket · postgresql · pgvector · google-cloud-run ·
google-cloud-sql · google-cloud-build · google-cloud-secret-manager ·
docker · html/css

Built With

Share this project:

Updates

Submission history