Hey Doug

Picture of physical hardware circuit
Electrical circuit diagram (RPi instead of Arduino)
Tinker circuit diagram (RPi instead of Arduino)

Inspiration

Modern AI tools like Cursor or Copilot are primarily optimized for outputting code rather than building the understanding of the user. While many tools try to include a "learning mode" in their tools list, we believe that something more needs to be done. There is a massive gap for students learning to code in areas where English isn't their first language or don't have enough money for private tutoring. While these aforementioned modern AI tools may help, a physical tool that a student can talk to using speech will help retain memory a lot clearer than a text based computer tool.

What it does

Doug the Debugging Dog is a multilingual, hardware-integrated AI teaching assistant. We built a physical device that acts as a one-to-one private tutor. Students press a button and verbally explain their problem in their native language (While only English and Spanish are supported on this version, Doug can integrate many more by demand). Doug then responds, explaining the solution by highlighting lines in the user's code editor as well as speaking to the user in the same language about what they can do about the problem.

How we built it

Physical Component:

Raspberry Pi for the processing power
ElevenLabs for the Speech to Text and Text to Speech
Breadboard+Resistors+Button+Switch for checking when user is talking (push to talk) and which language the user wants

VS Code Extension/IO Processing Component:

Gemini for converting input to output
Javascript for the extension
DeepL for text translation

Challenges we ran into

Audio: We found it hard to find a suitable input and output source for the audio
Bluetooth: The Raspberry Pi's Bluetooth Driver needed to reboot causing hours of lost testing time with the Pi due to us not having a backup wired keyboard connection.

Accomplishments that we're proud of

Working through debugging the Bluetooth Driver issue

What we learned

We learned that working with different forms of input and output for Raspberry Pis are hard.
We learned how much work can get done once we find ways to parallelize the tasks
We learned how to work with TinkerCAD to work with the breadboard and physical component layouts

What's next for Hey Doug

We want to add more language support for Hey Doug as well as a better system for audio input and output. Due to the limited supply of hardware on hand, there was only so much we could do in terms of creating a language selector on hardware and deal with microphones and speakers. With more time to scope out and buy components, we hope to improve the working prototype of Hey Doug to be a more genuine cohesive unit of a dog.