Voice copilot

Inspiration

I already had a basic voice-command app-launcher, built for myself: hotkey + voice command ("open firefox") launches the requested app. It served its purpose, but there's not much flexibility — one command, one action. For CUTC Transform, I wanted to make something that could genuinely comprehend and execute more complex requests based on natural language, to truly do what I want it to do: break complex intentions down into steps and execute them.

What it does

Voice Copilot is a desktop automation assistant for Linux, controllable by voice commands. Pressing Super+Z initiates a voice request: the assistant listens to what you say, and executes a command based on that.

• An app-local pattern matcher handles simple commands ("open firefox", "close terminal").
• Complex, multi-step requests ("take a screenshot, read the text on it, summarize it, and speak it back to me") are processed by breaking them down into individual steps using an LLM. Each step can make use of the previous result by substituting $prev_result, allowing actions to chain together.
• Unclear requests fall back to a guarded command generator, ensuring that ambiguous or unactionable requests can be retried or clarified by the user instead of failing silently.
• A live status panel displays the request transcript and current step, and a system tray button toggles listening on/off.

How I built it

The assistant's execution pipeline is relatively straightforward: upon pressing Super+Z, listen.sh records and transcribes the audio using whisper.cpp, and execute.py handles the request based on which of three broad categories it falls under:

• Fast lane — simple app launch/close commands are extracted via a pattern matcher and executed immediately, without LLM involvement.
• Planner + executor — for multi-step requests, planner.py sends the request plus a JSON schema of available actions to an LLM (hosted on Groq, running Llama 3.3 70B), which returns a JSON plan that's then executed by executor.py. The latter invokes real functions via actions.py, performing the requested actions and substituting $prev_result tokens where necessary.
• Raw-shell fallback — for anything the first two modes fail to handle, we ask the LLM to generate a single safe shell command that achieves the goal, or to return a clear refusal if the request can't be fulfilled.

Finally, app.py runs in the system tray, managing the status panel and the listening toggle.

Challenges I faced

The most challenging bugs to deal with were in the interaction between the different components — making sure the different modes interact safely and don't fail in ambiguous situations was a large part of the development process.

• A too-eager pattern matcher ate requests that should've gone to the planner. In an early version, my pattern matcher would immediately execute "open X" or "close X" commands, but it would falsely match those patterns even when embedded inside a longer, more complex request — bypassing the planner it should have deferred to.
• The status panel showed stale data when the fast lane handled a command. Since the fast lane never invoked the planner, it never updated the status panel — so if a command bypassed the planner, the panel kept showing whatever the previous command had left behind, which was misleading to look at mid-demo.
• Keybindings don't inherit your normal shell environment. My API key was exported in .bashrc, which worked fine when I ran the script manually from a terminal. But triggering the same script through my Cinnamon keyboard shortcut silently did nothing — because the keybinding runs in a minimal environment that never sources .bashrc, so the API key was never actually set. I only caught this by comparing manual runs against hotkey-triggered runs and adding a debug log; the fix was exporting the key directly inside the script itself rather than relying on the shell profile.
• LLMs aren't perfectly deterministic, and don't always follow instructions to the letter. I had a related issue with the raw-shell fallback: I instructed the LLM to return one exact string if it couldn't understand a request, but it would often return its own elaboration instead (e.g. "not recognizable what you said" rather than the exact phrase I asked for). Since my check was matching that one exact phrase, this caused silent failures whenever the model's wording didn't match precisely.

I had to be careful about the exact phrasing of my LLM prompts to avoid these kinds of issues, and to write more robust guards — checking for a pattern rather than an exact phrase (e.g. detecting any echo-style refusal in the raw-shell fallback, not just one specific sentence).

Lessons I learned

Tying an LLM to a deterministic pipeline introduces a class of errors that's unique to that combination. The most interesting defensive-programming opportunities showed up right at the interaction between the two, where my assumptions about how the LLM would behave turned out to be wrong. This project taught me that guarding against hallucinated or malformed LLM responses — with sensible fallback defaults, and broad pattern-based response guards instead of exact-phrase matching — is critically important once you're wiring probabilistic components into a deterministic system.

What's next

Some ideas I had but didn't have time to implement: adding speaker verification (so the assistant only responds to my voice), keeping the transcription model warm in the system tray process to reduce response latency, and abstracting the LLM call behind a common interface to decouple the app logic from any one specific provider.

AI tools used

I used Claude (Anthropic) throughout development — for architecture discussion, writing and debugging code, and drafting this submission's project story. I also briefly experimented with OpenCode, an open-source terminal coding agent, during development.

Built With

• bash
• llama
• piper
• pyqt5
• pytesseract
• python
• whisper.cpp