Spot The Bug

AI voice coding coach. Hunt bugs with progressive hints, pair program with shared screens and real-time code review, or solve coding problems with you. It talks, sees your code, and teaches.

Comment 1

About SpotTheBug

Inspiration

AI code assistants are everywhere — Copilot, Cursor, ChatGPT. And they're making developers worse. Studies show AI-assisted code contains 41% more bugs, because developers copy-paste without understanding what the code does. Junior developers are graduating without learning to debug. Senior developers are losing their edge.

I asked myself: what if AI could make developers smarter instead of lazier? Not by writing code for them, but by coaching them — the way a senior engineer would during a real pair programming session. Voice-first, conversational, Socratic. That became SpotTheBug.

What It Does

SpotTheBug is an AI coding coach that talks to you and sees your screen in real-time. It has three modes:

🔍 Bug Hunt — The AI generates a realistic buggy code snippet (grounded via Google Search to ensure it's based on real-world patterns). You read the code, think out loud, and the AI coaches you with progressive hints through voice conversation — it directs you to find and fix the bug yourself. When you fix the bug, a Google ADK agent evaluates the session transcript and scores your performance.

🤝 Pair Programming — Share your screen and select your local project folder. The AI sees your editor live via multimodal vision, reads your actual files through function calling, and focuses specifically on your local code changes — reviewing them file-by-file like a real tech lead doing a code review, all via real-time voice.

🧩 Problem Solve — AI generates a coding challenge (grounded via Google Search). You write your solution from scratch in a Monaco editor. Run tests in a real sandboxed environment (E2B). The AI coaches you through voice — asking about your approach, reviewing your logic, giving hints if you're stuck.

How I Built It

Architecture

Next.js 16 full-stack app deployed on Google Cloud Run with multi-stage Docker build. Prisma ORM with SQLite for session/user data. NextAuth for authentication.

Voice Engine (Core)

The heart of SpotTheBug is the Gemini Live API (gemini-2.5-flash-native-audio). Each mode has a dedicated React hook (useHuntVoice, usePairVoice, useProblemSolvingVoice) that manages:

  • WebSocket connection via @google/genai SDK
  • Real-time bidirectional audio streaming (PCM 16kHz, downsampled from native sample rate)
  • Session resumption with handle persistence — when the server times out, the client auto-reconnects with the same conversation context
  • Context window compression (200K trigger → 100K sliding window) for long sessions

Multimodal Vision (Pair Mode)

Screen capture at 1fps → JPEG encoding → sent as sendRealtimeInput({ video }). The AI literally watches the developer's screen and comments on what it sees. Combined with the readFile function calling tool, the AI can both see the screen AND read the actual file contents for accurate line-level code review.

Grounded Bug Generation (Hunt Mode)

3-phase pipeline using Gemini + Google Search:

  1. Generate — Search for real bug patterns in the target framework
  2. Validate — Search-verify the bug is actually a real bug with correct fix
  3. Structure — Convert to typed JSON (no search tools in this phase to enable JSON mode)

Grounded Fix Evaluation (Hunt Mode)

2-phase pipeline: Gemini evaluates the developer's fix using Google Search to verify the underlying bug mechanism, then structures the result into JSON.

Pre-Session Code Review (Pair Mode)

Before the voice call starts, the system analyzes the developer's selected files and generates a list of findings (severity, line numbers, explanations). These findings are injected into the AI's system instruction as CONTEXT YOU ALREADY KNOW — so when the voice session begins, the AI already knows every issue and can guide the discussion file-by-file.

Post-Session Evaluation (Hunt Mode)

Google ADK (Agent Development Kit) evaluates the session transcript after each Bug Hunt round. It analyzes what bugs the developer detected, where they struggled, and produces structured performance feedback.

Challenges I Faced

1. Gemini Live API + Google Search — Can't Use Both With JSON Mode

The biggest architectural challenge. Google Search grounding and JSON response mode (responseMimeType: "application/json") are mutually exclusive in the Gemini API. I couldn't generate a grounded, structured bug in a single call. The solution was the multi-phase pipeline: Phase 1 uses free text + Google Search for grounded generation, Phase 2 uses free text + Google Search for validation, Phase 3 uses JSON mode (no search) for structuring. This pattern repeats across bug generation and fix evaluation.

2. Session Timeout and Reconnection

The Gemini Live API closes WebSocket connections after a server-side timeout. Mid-conversation disconnects were killing the user experience. I implemented session resumption — the server sends GoAway warnings and resumption handles. When the connection drops, the client automatically reconnects with the same handle, and the conversation continues seamlessly. Combined with context window compression (sliding window), sessions can run for extended periods without losing context.

3. Screen Vision Timing

In Pair mode, if the AI starts talking before seeing the first screen frame, it hallucinates about code. I solved this with a first-frame gate: the greeting prompt is only sent AFTER the first JPEG frame is successfully delivered to the model. The onFirstFrameRef callback pattern ensures the AI always sees the screen before it speaks.

4. Function Calling During Live Voice

Making the readFile tool work during an active voice session was tricky. The AI sends a toolCall, the client reads the file from the user's local filesystem (via File System Access API + FileSystemDirectoryHandle), and sends the response back — all while the voice stream continues. Error handling had to be bulletproof because the WebSocket could close between the tool call and response.

What I Learned

  • Gemini Live API's native audio is a game-changer for educational AI. The voice quality and interruption handling make conversations feel genuinely human — no TTS artifacts.
  • Google Search grounding transforms AI-generated content from "plausible fiction" to "verified reality." Every bug in SpotTheBug is based on real patterns, not hallucinated ones.
  • Multi-phase pipelines are the practical solution when you need both grounding AND structured output from Gemini. Embrace the pipeline, don't fight the API constraints.
  • Session resumption with context compression makes long-running Gemini sessions viable in production. Without it, the 10-minute timeout would make the product unusable.
  • The Socratic method works when backed by real technology — AI that can see, hear, and read code simultaneously creates a coaching experience that text-only tools can't match.

Built With

Share this project:

Updates

Submission history