Sus.AI - Interrogation Game

Inspiration

I wanted to push the boundaries of what AI games could be, moving beyond traditional text-based adventures. Inspired by classic detective games like L.A. Noire and Phoenix Wright, I envisioned a truly immersive experience where you don't just select dialogue options—you actually speak to the suspects. With the advent of fast, multimodal AI, I realized I could build a game where players use their real voice and physical objects in the real world to solve virtual crimes.

What it does

Sus.AI puts you in the shoes of a lead detective. You choose from 22 unique cases ranging from stolen diamonds to missing cookies. Using your microphone, you interrogate dynamic AI suspects in real-time. The core twist is the "Impromptu Camera Mechanic": suspects can see you through your webcam. If you hold up the correct physical object (like a coffee mug, a watch, or a set of keys) that contradicts their alibi, they will panic and confess, winning you the game. It also features dynamic, context-aware question suggestions if you ever get stuck.

How I built it

I built the frontend using React, TypeScript, and Tailwind CSS for a sleek, immersive UI. The core engine relies heavily on the new Google Gemini Live API (gemini-2.5-flash-native-audio-preview). I used the browser's Web Audio API and MediaDevices to capture and stream raw PCM audio and video frames over WebSockets directly to the model. To help players, I integrated a secondary background process using gemini-3-flash-preview that reads the rolling transcript and generates punchy, context-aware interrogation suggestions.

Challenges I ran into

Handling real-time, bidirectional audio and video streaming in the browser was a major technical hurdle. I had to carefully manage WebSocket lifecycles, audio buffering, and reconnection logic to ensure a seamless experience. Another significant challenge was prompt engineering: getting the AI to stay strictly in character, react naturally to visual cues from the webcam, and reliably trigger the internal "confess" function without breaking the illusion required extensive tuning and testing. I also had to implement exponential backoff strategies to gracefully handle API rate limits.

Accomplishments that I'm proud of

I am incredibly proud of the "Impromptu Camera Mechanic." The moment you hold up a real-world object to your webcam and the AI suspect instantly reacts, stammers, and confesses is pure magic—it bridges the gap between the physical and digital worlds perfectly. I'm also proud of the robust error handling and reconnection logic that keeps the game stable even during network hiccups.

What I learned

Building Sus.AI deepened my knowledge of WebSockets and real-time media streaming (handling raw PCM audio data). I learned a massive amount about the nuances of prompting multimodal LLMs, specifically how to balance system instructions to handle both conversational roleplay and strict tool-calling (function calling) simultaneously.

What's next for Sus.AI - Interrogation Game

I plan to introduce a full "Campaign Mode" where cases are linked by an overarching narrative. I also want to add a progression system where detectives earn ranks and unlock harder cases. Eventually, I'd love to explore multi-suspect interrogations (where suspects can contradict each other) and potentially a multiplayer co-op mode where two detectives play "good cop, bad cop."

Built With

• css
• googlegeminiapi
• googlegeminiliveapi
• lucidereact
• react
• tailwind
• typescript
• vite
• webaudioapi/mediadevicesapi