How we built it
- Inference: a 235B-parameter model on a single GPU, built on the open-source llama.cpp inference engine.
- Orchestrator (Rust): the core of DuckyCode. It turns the model's output into natural speech, live typing and edits across files, and keeps the experience smooth and responsive.
- VS Code extensions (TypeScript): live typing, file navigation, highlighting, and a voice-first panel with a mic, transcript, diagrams and Keep/Undo. The voice and mic stay on your own machine even when the model runs elsewhere.
- Speech: a natural neural voice and speech recognition.
- Testing: a simulated user that plays out realistic conversations against the real model, which we used to tune the experience.
Challenges we ran into
- Performance: getting a model this large to run well on limited hardware alongside the voice.
- Natural interruptions: making it stop and pick up again seamlessly when you cut in, mid-sentence and mid-line.
- Getting the model to actually listen: making it follow your direction, answer questions without changing course, and finish the whole task across files.
- Voice in a real room: letting you talk over it without it mishearing itself.
- Infrastructure: cluster outages, hardware differences, and a very large model to load. We got startup down from minutes to seconds.
Accomplishments that we're proud of
- A 235B model running usefully on one GPU. A 480B coder model also ran at about 3.6 tok/s.
- Our projections suggest it could run at around human typing speed on a 32 GB-RAM laptop.
- Interruptions feel natural: the voice responds in about 1–2 seconds and carries on where it stopped.
- Out-of-order, multi-file coding that narrates why, like a person would.
- Guided codebase tours that follow how the code actually runs, across files.
What we learned
- Latency is a UX problem as much as a systems problem. Matched to the right interaction, a slow model feels fine.
- The human, not the hardware, can be the real bottleneck. Designing for that opens up much bigger models on much smaller machines.
- Prompting alone doesn't make a model behave. The details of the surrounding system matter as much.
- Testing conversations with a simulated user was worth it. It caught problems we'd never notice by hand.
What's next for DuckyCode
- DuckLLM, our own inference work, to push performance further on edge hardware.
- Running fully on a laptop, with no cluster.
- More natural voice interaction.
- Bigger codebases and longer guided tours.
- A learning mode for teaching beginners.
Built With
- cuda
- llm-inference
- pytorch
- rust
- typescript
