CodePilot

A voice-first coding platform that teaches through personalized lessons, live conversation, and hands-on programming exercises.

Comment

Inspiration

Learning to code can feel intimidating, especially when students are left alone with documentation, error messages, and unfamiliar tools. We wanted to create an experience that feels more like learning alongside a patient tutor.

CodePilot was inspired by the idea that coding education should be conversational, accessible, and hands-on. Instead of relying on a traditional chatbot, CodePilot uses a voice-first learning experience where students can listen, speak, read live transcripts, and practice code in the same workspace.

What it does

Inspiration Learning to code can feel intimidating, especially when students are left alone with documentation, error messages, and unfamiliar tools. We wanted to create an experience that feels more like learning alongside a patient tutor.

CodePilot was inspired by the idea that coding education should be conversational, accessible, and hands-on. Instead of relying on a traditional chatbot, CodePilot uses a voice-first learning experience where students can listen, speak, read live transcripts, and practice code in the same workspace.

What it does CodePilot is a personalized coding education platform that combines:

Structured programming lessons A voice-enabled tutor Live speech transcription An AI-style teaching whiteboard Interactive coding challenges A Monaco code editor In-browser Python execution Progress tracking Personalized course dashboards Students can explore lessons, review explanations and examples, and start an interactive learning session. During a session, the tutor speaks using browser speech synthesis and listens through browser speech recognition.

The tutor receives educational context including the current language, lesson, challenge, progress, conversation, and code being written. Students can run Python directly in the browser and view the output without leaving CodePilot.

How we built it

We built CodePilot with React, TypeScript, Vite, and Tailwind CSS.

Firebase provides authentication and stores user profiles and learning preferences. React Router handles protected pages, dynamic courses, lesson sessions, onboarding, and authentication routes.

Monaco Editor powers the coding workspace, giving students an experience similar to Visual Studio Code. Python programs run directly in the browser through Pyodide.

The voice experience uses the browser’s native Speech Recognition and Speech Synthesis APIs. We created a conversation state machine with states for speaking, waiting, listening, thinking, and responding.

We also created reusable educational services:

TutorContextService tracks the active lesson, challenge, progress, code, and conversation. ConversationManager maintains recent tutor and student exchanges. AiTutorEngine generates context-aware placeholder guidance without requiring a paid external AI service. SpeechRecognitionService isolates browser speech recognition so it can later be replaced by another provider.

Challenges we ran into

One major challenge was making the voice conversation feel natural. A tutor should not continuously speak—it must explain something, ask a question, wait, listen, process the response, and continue teaching. Building this required careful coordination between speech synthesis, speech recognition, transcript updates, and conversation states.

Another challenge was keeping the tutor aware of everything happening in the session. We needed the educational context to update whenever the student changed lessons, edited code, progressed through the course, or spoke to the tutor.

Running code safely was another challenge. We used Pyodide to execute Python inside the browser, avoiding the need for a code-execution backend during this stage.

We also had to handle browser compatibility, microphone permissions, denied permissions, unsupported speech-recognition APIs, asynchronous loading, and TypeScript safety without allowing the interface to crash.

Accomplishments that we're proud of

We are proud that CodePilot combines an entire learning workflow in one cohesive experience.

Students can:

Learn a programming concept Hear the concept explained Read accessible live subtitles Ask questions using their voice Write code in Monaco Run Python in the browser Review the program output Continue learning from the same workspace We are also proud of the platform’s modular architecture. The tutor, transcript, conversation engine, lesson context, code execution, and editor are separate reusable systems that can evolve independently.

What we learned

We learned that building a voice-first experience requires more than adding speech recognition. A natural conversation needs clear states, interruption handling, transcript synchronization, permission management, and accessible visual feedback.

We also learned how to integrate Monaco Editor, execute Python through WebAssembly, build dynamic lesson architectures, manage protected Firebase routes, and keep educational context synchronized with a student’s actions.

Most importantly, we learned that an effective coding tutor needs to understand more than a question. It needs lesson context, student progress, recent conversation, and the code currently being written.

What's next for CodePilot

Our next goal is to connect the existing AiTutorEngine to a capable language model when sustainable hosting becomes available. Because the context and conversation architecture already exists, this can be added without redesigning the interface.

We also plan to add:

Persistent lesson progress More complete courses and programming languages Secure execution for additional languages Automated code feedback Personalized learning paths Quizzes and assessments Improved accessibility options Better voice selection and controls Project-based learning Tutor-generated examples and whiteboard content Mobile and cross-browser improvements Our long-term vision is for CodePilot to feel like a personal coding instructor that is available to every learner.

Built With

Share this project:

Updates