Inspiration

We’ve all stared at an unfamiliar app and thought, “What do I click next?” Searching for help often means leaving the task, sorting through tutorials, and figuring out which instructions match your screen.

We wanted help to meet you where you are. That inspired Coach: a companion you can ask about your screen, through voice or text, that explains the next step and points you toward it.

What it does

HeyCoach is a native macOS screen companion. You can hold a push-to-talk shortcut and say “Hey Coach,” or open a text box and type your question.

It captures your screen for the interaction, sends it to Gemma, and uses the response to explain what you’re looking at and point to relevant controls. An optional computer-use mode can propose a single click, double-click, or right-click and ask for confirmation before acting.

Our goal is to make unfamiliar software easier to navigate—one clear step at a time.

How we built it

We started with the MIT-licensed Clicky project and adapted it into Coach.

SwiftUI and AppKit power the native menu-bar interface, text-input panel, and cursor overlay.
ScreenCaptureKit provides screenshots for visual context.
Gemma, accessed through the Gemini API, interprets screenshots and generates responses.
A Node.js backend on DigitalOcean handles model requests, streams responses, and keeps provider credentials off the desktop app.
macOS Accessibility APIs and CoreGraphics support the optional confirmed-click feature.

We also added native macOS speech fallback for typed answers when the external voice service is unavailable.

Challenges we faced

Making computer use trustworthy. A coordinate alone is not enough: a window can move between the proposal and confirmation. We added checks to compare the target’s Accessibility identity and geometry before clicking, and to refuse targets that cannot be verified.

Combining voice and text input. Both share one conversation, but their lifecycles can overlap. We had to handle cancellation, stop interrupted speech, and prevent closing the text box from resetting an active voice interaction.

Working across Windows and macOS. Development and backend testing happened on Windows, while the native interface and computer-use features required a teammate’s Mac. Clear setup instructions and honest verification boundaries were essential.

Deploying alongside an existing application. We hosted the backend on an existing DigitalOcean server using a separate Node runtime, service, and HTTPS configuration.

What we learned

We learned that a useful AI companion needs more than a good model response. It needs clear feedback, reliable cancellation, secure credential handling, and careful boundaries around actions.

Code review also caught issues that a happy-path demo could miss: stale click targets, invalid screen coordinates, overlapping interactions, and typed content entering analytics. Fixing those issues became an important part of building HeyCoach.

What we’re proud of

We brought screen understanding, pointing, typed interaction, and confirmed computer use into one companion. Our deployed backend passed 49 automated tests, and we verified a real authenticated Gemma response over HTTPS.

Native macOS build and end-to-end interaction testing remained a separate verification step; we distinguish that from the backend checks we completed.

What’s next

We want to add a quiet mode with visible text answers, improve shortcut customization, and expand Mac testing across apps and multiple displays.

Longer term, HeyCoach could help people work through unfamiliar tasks with a persistent checklist—remembering where they stopped and helping them resume with one manageable next step.