Inspiration
What it does
Bud is a voice-first assistant that lives as a translucent overlay on top of the desktop and sees what you see. Say "Hey Bud", ask a question naturally, and Bud answers out loud and draws a box around the exact element it is describing.
The key innovation is that Bud is local-first and smart about where work happens:
- Screen understanding and question classification run entirely on-device on a bundled Qwen3-VL-4B vision-language model served by llama.cpp on the GPU, and your screen never has to leave your machine for visual tasks.
- Only general-knowledge questions are routed to the cloud (a configurable Nebius model), so the common case stays fast and private.
- A lightweight classifier decides between the two paths automatically and defaults to screen context whenever the question is ambiguous.
- Everything is provisioned on first run: runtime, model weights, and speech models are downloaded automatically and started on free local ports, so there is no manual model setup.
How we built it
- Electron was used for the creation of the desktop application, combined with a React frontend.
- Python is used for the STT and TTS backend.
- The STT is powered by openwakeword for wake word detection and Moonshine (small streaming 123M).
- TTS is powered by Pocket TTS's recent 18M model.
- LiveKit WakeWord is used for the training of the wake word "hey bud" on both positive and negative clips, minimizing false positive activations.
- Qwen-3VL-4B is used for the visual understanding of the screen, running on a LLaMA CPP backend.
- Agentic coding is done using mostly OpenCode free models.
Challenges we ran into
- When asking it to translate images to a target languages, it often failed to put the translation in the "copy" button, even though explicitly asked. At first the idea was to create custom prompt presets that can be activated for specialized responses. This ended up becoming overcomplex for general usage and was still hard to tweak correctly. It seemed that many system prompts were hard to get right for the correct copy value. It seemed like the copy label of the JSON response had great intentions to just extract the original text. A hidden label that writes a short description of the intent helped producing correct copy values, without needing to switch to a thinking model adding extra latency.
- Qwen-3VL-4B takes a significant portion of the VRAM usage. To prevent adding lots of additional resources, we needed to work with lightweight STT and TTS models, while still giving great results.
Accomplishments that we're proud of
- I am satisfied with how the overall look of the app turned out.
- It took some time to get all part together for voice assistance, but the initial version that worked was really satisfying.
What we learned
- The orchestration of multiple models to achieve voice assistance.
- Working with multiprocess applications and its communication.
- System prompting and using JSON Shemas for consistent expected output.
What's next for Bud
- Allow bud to act on the response for computer automation. Potentially testing with vl models fine tuned or specialized in GUI might give better accuracy for consistent automation.
- The current build focussed on purely Windows platforms. Future steps could be also adding Linux and Mac support.
- Current Moonshine STT model have the potential to transcribe conversations in real-time, identifying speakers separately, Combined with LLM, we can combine this with translation.
Built With
- electron
- llm
- python
- react
- stt
- tts
- typescript
- vlm
