Inspiration
Using a computer takes thousands of tiny movements a day like being able to click, drag, type, scroll, or even surf the web. It's like second nature to a majority of people.
However, those with injuries, vision impairments, or severe mobility loss may find it difficult to complete everyday tasks. Each of those movements cost time, effort, and possibly pain. Accessibility tools exist, but many of them make you learn a whole new set of commands.
We asked ourselves: What if you could just say what you want, like asking a friend? Then we remembered Clippy, the paperclip assistant from old Microsoft Office, and decided to bring it back. This time it’s a friendly companion, a bear, and it actually does the task for you.
Our voice-first Windows companion allows people who aren't able to comfortably use a mouse or keyboard run their computer by simply talking!
What it does
GrizzHelper is a bear companion that sits on your Windows 11 desktop with a speech bubble. Hold F9 (or press Talk) and speak, or type a request like "Open Notepad and write a note: buy milk" or "Open my Downloads folder and open the newest PDF."
An AI plans the steps and GrizzHelper carries them out in real Windows apps, then tells you what it did, in the bubble and out loud. Before a risky action, like deleting text or closing a window, it stops and asks yes or no. Ask "What's on my screen?" and it describes what it sees.
Can't use your hands? Hands-free mode tracks your head with the webcam, entirely on your laptop. Hold still on a button for two seconds to press it, including Talk and the yes/no buttons, so you can finish a whole request without touching the keyboard or mouse.
How we built it
Brain: OpenAI's open-weight gpt-oss-20b, served with vLLM on an NVIDIA Brev GPU and called with the openai Python SDK using tool calling. We wrote our own agent loop from scratch with no frameworks: six small tools and a cap of 8 steps. Hands: A Python backend. pywinauto (Windows UI Automation) finds and focuses app windows. The keyboard library and clipboard paste type text and press shortcuts. Eyes: Gemini vision plus mss screenshots answer "What's on my screen?". We only send a screenshot when asked, and retry with backoff on rate limits. Voice: ElevenLabs Scribe for speech-to-text, primed with app names it might mishear. ElevenLabs Flash for text-to-speech, streamed so the bear starts talking before the audio finishes downloading. Face: An Electron mascot in plain JavaScript, talking to the backend over a WebSocket on localhost, with idle, thinking, talking and confirm states. Hands-free mode uses MediaPipe Face Landmarker in the window: where your nose sits relative to your eyes moves a pointer, and a two-second hold clicks.
Challenges we ran into
- Tool calling on a self-hosted model: We had to set up vLLM with the right flags for gpt-oss, and tool_choice="auto" is the only option it supports. So we kept our tools small (6–8) with tight descriptions to keep the model on track.
- Rate limits and latency: We’re on Gemini’s free tier, so we only send screenshots when we have to. We also had to keep voice responses fast enough to feel like a conversation.
- Typing reliably: Typing one key at a time dropped letters ("GrizzHelper" came out "rrrr") while our push-to-talk hotkey listener was running. Now we paste text through the clipboard and put the user's clipboard back afterwards.
- Focus: Clicking the mascot takes keyboard focus, so "close Calculator" once closed the bear itself. Every keyboard tool now re-focuses the app it last opened.
- Steering the model: A true/false "new note?" argument was wrong about 1 time in 3. A one-line hint in the tool's result worked almost every time.
- Safety: An AI that presses keys on your PC needs guardrails. It can only open apps on an allowlist (no terminals), only open safe file types (never .exe), and risky keys like Delete or Alt+F4 need a yes.
- A $150 GPU budget: We ran a single Brev instance and watched the clock.
Accomplishments that we're proud of
- Voice in, real action in a Windows app, voice out, end to end on a self-hosted open-weight model with a loop we wrote ourselves.
- A fully hands-free path: head pointing plus a two-second hold works Talk and the yes/no confirmation.
- Safety built in from the start: allowlisted apps, safe file types, confirmations, and a backend that only listens on localhost.
What we learned
We learned how a model actually turns language into actions (tool schemas, tool-call IDs, step limits). Also, we discovered how Windows exposes its UI to assistive tech and how it serves on an open-weight LLM on a cloud GPU. That accessibility design means planning for failure: when the AI is unsure, the bear should ask.
What's next for GrizzHelper
More accessibility features, as of right now it's a hopeful prototype. We would expect to be able to implement clicking buttons by name with UI Automation, using Gemini to find controls UI Automation can't see, mouse drags for apps like Paint, more apps like Chrome, and a head pointer that reaches the whole screen instead of just the bear's window.