Ghostline

Inspiration

Most AI voice demos still feel like advanced chatbots with audio; they are turn-based, lack spatial awareness, and bluff or hallucinate when they can't understand the physical world. I wanted to build something that breaks the text box paradigm. I was inspired to create a live, camera-aware, interruptible experience that feels less like chatting with an AI and more like being managed by a real operator during an active, high-stakes incident. Thus, the idea for a paranormal containment hotline was born.

What it does

Ghostline is a live paranormal containment hotline built for the Live Agents category. When a user calls in, they speak with The Archivist, a calm, procedural containment operator, in real time.

Rather than a free-form sandbox, Ghostline is a strict, deterministic experience. The operator asks for camera and microphone access during the call, assesses the user's room, and assigns a curated sequence of physical containment tasks (e.g., closing boundaries, increasing light, drawing marks).

Crucially, the agent evaluates real-time camera frames during deliberate Ready to Verify checks. If the system cannot see the required evidence, the Archivist honestly reports uncertainty, offers alternative tasks, and guides the user to a secure resolution, ending with a complete case report.

How I built it

Ghostline is built around Gemini Live on Vertex AI (gemini-live-2.5-flash-native-audio). Frontend: Built with React, Vite, and TypeScript. It handles real-time media capture, audio streaming, UI state, and features a visible grounding HUD so the user always knows the Agent's internal state. Backend: Powered by Python, FastAPI, and WebSockets. It acts as the bridge to Gemini Live and manages the strict, deterministic state machine that governs task progression, capability failures, and Ready to Verify visual checkpoints.

Cloud Architecture: Hosted firmly on Google Cloud Run using Artifact Registry and Cloud Build. We emit structured operational events to Cloud Logging to prove the backend's cloud execution and maintain system reliability.

Challenges I ran into

Barge-in Mechanics: Balancing live audio with a deterministic flow was difficult. I had to ensure that when a user interrupted (barge-in), Gemini instantly stopped speaking, flushed its audio queue, and gracefully returned to the current task without breaking the state machine.

Staged Verification vs. Continuous Bluffing: I initially experimented with continuous frame processing, but found it led to AI hallucination (the AI would confidently claim to see things that weren't there). I shifted to a deliberate "Ready to Verify" staging model where the agent only makes high-confidence checks when the user holds still, greatly improving reliability and trust.

Accomplishments that we're proud of

The Persona: I successfully created The Archivist, a character that feels observant, controlled, and professional, rather than melodramatic and campy.

Handling Failure Gracefully: Ghostline does not have dead ends. If a user cannot perform a task, the Agent accepts the constraint, adapts, and selects a substitute task from its curated library while recording the failure honestly on the HUD.

The Grounding HUD: I built a highly functional UI that displays exactly what the agent is thinking, proving to the user that the system is operating logically and grounded in reality. What I learned

Barge-in is everything: How an agent handles being interrupted is the single biggest factor in whether it feels live or scripted.

Honesty builds trust: Staged verification where the AI admits "The frame is too dark to confirm" builds vastly more trust than a system that bluffs continuous certainty.

Constrain the domain: Bounding the AI to a curated library of tasks and a strict state machine produces a far superior product compared to free-form generation.

What's next for Ghostline

I want to expand the task library and integrate cloud-backed persistence (like Firestore) so returning callers can have their previous case files referenced organically by The Archivist. I'm also exploring how this exact architecture—operator-led, camera-aware, state-machine validated—can be adapted into enterprise use cases like real-time industrial safety inspections, remote medical triage, and live maintenance support.

Built With

• artifact-registry
• fastapi
• gemini
• google-cloud
• google-cloud-build
• google-cloud-run
• python
• react
• typescript
• vertex-ai
• vite
• websockets