EchoCode

Inspiration

I've always enjoyed using Claude or GPT for learning about code bases but some times I wish I could just talk to the model and have it respond similar to how a asking another engineer would explain how a file or codebase works and this project aims to help with that.

What it does

EchoCode is a real-time voice-activated code co-pilot that provides accurate, audio-based assistance based on your local codebase. It accomplishes this via three key pillars:

Real-Time Audio Pipeline: Converts your speech to text (Deepgram), processes the query (Claude 3.5 Sonnet), and converts the answer back to speech (Fish Audio) in under 4 seconds.

Semantic Context: A custom CLI continuously syncs your code files to a backend vector index, enabling EchoCode to use semantic search to send Claude only the most relevant code context for each query.

Hands-Free Workflow: You ask questions verbally, and the answer is streamed back via smooth, gap-free TTS audio playback handled by Media Source Extensions (MSE) in the UI.

How we built it

copilot-server (Backend): An Express.js WebSocket server managing the entire flow. It orchestrates Deepgram STT, streams the query to Claude 3.5 Sonnet, and streams audio back from Fish Audio TTS using async generators for memory efficiency.

copilot-cli (File Watcher): A custom Node.js CLI using chokidar that watches the user's local directory and uses HTTP endpoints to keep the backend's file cache and vector index updated.

copilot-ui (Frontend): A Next.js/React UI that captures microphone input and uses Media Source Extensions (MSE) for reliable, gap-free playback of the streamed TTS audio.

Challenges we ran into

Originally this project was not supposed to exist. It was a last minute switch because my original project idea of a real time AI tutor that would guide you through problems using Livekit and Claude and letta was out of scope for this weekend. So it was difficult to swap project ideas last minute, however I wanted to keep some elements of the original idea.

Accomplishments that we're proud of

I'm proud that I was able to get a working demo of this idea in action within such a short time frame I started at 6PM of October 25th.

What we learned

I learned a lot about Text to speech, and websockets to hold connections. I also learned more about semantic search, and how we can search for certain things with just the idea of what we wanted. It was also a great learning opportunity to learn about Fish Audio, DeepGram, and Anthropic.

What's next for EchoCode

• Deployment
• IDE Integration: Develop a VS Code or JetBrains extension to embed the UI directly, allowing contextual actions on code selections.
• Conversation History: Debug and finalize the conversation history feature to maintain multi-turn context throughout the session. hopefully with letta 👀
• CLI Robustness: Implement file deletion sync to ensure the semantic search index accurately reflects the live codebase.
• WebRTC Migration: Explore migrating microphone input to WebRTC for even lower latency audio streaming.

Built With

• claudeapi
• deepgramapi
• express.js
• fishaudioapi
• next.js
• react
• typescript