SideKick

Your voice-first coding colleague: narrates real diffs, flags bugs in a second voice, answers spoken Q&A, and sketches live.

Comment

Disclaimer: Detailed information about how we've used every single sponsor can be found in our project README (https://github.com/Mallika1405/SideKick#how-each-sponsor-is-used)

Inspiration

Have you ever vibe coded at the last minute, watched Claude Code rewrite half your project, and had no idea what it actually changed? Undergraduate researchers increasingly rely on AI coding agents like Claude Code to write data analysis scripts—and then have to defend those results to a PI, a labmate, or a peer reviewer.

We asked: What if you had an expert software engineer sitting right beside you at a whiteboard—watching the actual code diffs as they happen, explaining what changed in plain spoken English, calling out bugs in a distinct voice, and sketching out the architecture when a visual helps?

What it does

Sidekick is a voice-first companion for AI-assisted coding that lives entirely inside VS Code. While Claude Code (or you) edits a repository, Sidekick operates across four independently toggleable modes in the editor status bar:

  1. Narrate: Speaks each code change in plain, concise language—generated strictly from the actual file diff and surrounding context, never from Claude Code's self-reported summary. As it speaks, it highlights the exact lines being discussed and jumps your editor to them.
  2. Listen (Spoken Q&A with Barge-In): Answers spoken questions about the codebase with exact file and line citations. Powered by turn-based speech-to-text, you can interrupt Sidekick mid-sentence (Cmd+Alt+T or hands-free Always-On Mic) just like a real conversation.
  3. Review: Passively monitors diffs and flags likely bugs aloud in a distinct second voice while placing inline editor decorations (red = bug, orange = warn, blue = info) completely separate from standard language-server squiggles. To avoid nagging, only bug severity is spoken aloud.
  4. Live Whiteboard: Opens a side-panel right next to your code with three synchronized views that switch automatically as events happen (with a 20-second pin when clicked):
    • Map: A high-level architecture graph built incrementally from JSON patches whenever components are added.
    • Detail: A deterministic call/import/definition graph parsed directly from Python and TypeScript/JavaScript AST/regex structures—zero hallucinated structure, following either the narrator's focus or your cursor.
    • Sketch: Hand-style diagrams (rough.js boxes, notes, tables, and clearly labeled schematic plots) drawn step-by-step in sync with the sentence being spoken.

How we built it

We designed Sidekick around a local Node.js + TypeScript hub (server/) that bridges Claude Code HTTP hooks, our VS Code extension (extension/), a Vite + React whiteboard console (console/), a native macOS Swift audio helper (extension/native/), and our sponsor stack:

  • Claude Code Hooks & Editor Capture: When Claude Code edits files (via PostToolUse, UserPromptSubmit, and Stop hooks—plus directory re-scanning on Stop to catch files written via shell commands) or when a human types in VS Code, the server diffs the file against its saved snapshot and groups edits after ~1.5s of quiet.
  • Anthropic (Claude Haiku 4.5 / 5.5 & Sonnet 4.5 / 5.5): Every language task is grounded in the real diff or retrieved code. Fast narration (narrate.ts) and strict-JSON code review (review.ts) run on Haiku for low latency, while deep codebase Q&A (answer.ts) and incremental whiteboard graph patches (graph.ts) run on Sonnet. Extended thinking is disabled on Haiku and set to between_tools on Sonnet so token budgets go directly toward immediate output.
  • Cartesia (Sonic TTS & Ink-2 STT): Cartesia powers our entire voice pipeline over server-side WebSockets. We stream LLM output sentence-by-sentence into Cartesia TTS (sonic-3.5-2026-05-04) using a single context_id per utterance (continue: true) across two distinct voices (Narrator and Reviewer). For input, Cartesia's ink-2 turn-based WebSocket (16 kHz PCM s16le) gives us turn.start for instant barge-in context cancellation, turn.eager_end for speculative retrieval, and turn.end to commit the question.
  • Moss (In-Process Semantic Retrieval): Code is chunked at def/class/function boundaries with line numbers (chunker.ts) and indexed in-process via MossRetriever (moss-minilm on client.session()). Every file save, Claude edit, narration, and graph summary dynamically updates the index so spoken questions ("where do we turn the results into a picture?") retrieve the top-5 semantic chunks in ~1 ms (with a TF-IDF keyword fallback).
  • Convex (Reactive State & Persistence): Sessions, edits, narrations, issues, and architecture graphs are stored in Convex (convex/schema.ts, convex/sidekick.ts) behind a clean Store interface. The React whiteboard subscribes via useQuery, updating the visual graph reactively as diffs arrive.
  • Notability (From Handwritten Spec to Code): Our entire architecture and four-mode status bar started as a handwritten Notability ideation note complete with architecture sketches and an embedded voice discussion (https://notability.com/app/note/ae96014d-8664-420a-85fa-d5a0654ee037).

Challenges we ran into

  • VS Code webviews cannot resolve getUserMedia to access the microphone. We engineered and compiled a native macOS Swift helper (sidekick-audio.swift) that captures mic input and plays audio natively, communicating seamlessly with the extension.
  • We noticed Claude Code was occasionally writing or modifying files via bash/shell commands rather than standard edit tools, bypassing file hooks. We broadened our hook listeners and added an automatic project tree re-scan on Stop and after shell tool invocations.

Accomplishments that we're proud of

  • 100% on our Narration Eval: We built a reproducible evaluation suite (npm run eval) that replays 15 scripted research-code diffs (data leakage before train/test split, removed Bonferroni caps, disabled Welch t-test corrections, Random Forest tree count drops from 200 to 5, plus benign controls) against a 60-file repo. Sidekick scored 100% accurate in 14–26 words per narration—proving it catches subtle methodological shifts without inventing drama on benign edits.
  • Sub-Millisecond Semantic Search Updates: On our 60-file demo repository (104 chunks), Moss queries run in ~1 ms and single-chunk live re-indexing on file save takes ~7 ms.
  • Zero-Hallucination Detail View: Because the Detail whiteboard parses Python and TS/JS definitions and references deterministically rather than asking an LLM to draw file internals, researchers can trust the call graph 100%.

What's next for Sidekick

To take Sidekick from a hackathon project to a daily tool for vibe coders:

  1. Publish the extension to the VS Code Marketplace.
  2. Port our Swift audio helper (sidekick-audio) to Windows and Linux so native voice works across all OS environments, and verify compatibility across Cursor and other VS Code forks.
  3. More customization options for the sketching, reviewing and narration tools.

Built With

Share this project:

Updates

Submission history