Inspiration
It’s the night before your first day at a new job. Someone on the team mentioned an app they use, so you open it up to get a little practice. A few minutes later, you’ve got a YouTube tutorial open beside it. You pause the video just as the person clicks, switch back to your window, and look in the same spot. The button isn’t there. You send a screenshot to AI and ask where to click. It tells you the name of the button. Fair enough, but you still don’t know which one it is. We know that feeling from school, too. Sometimes a class assignment is the first time you’ve even opened the tool you’re supposed to use. MIT has a whole course called The Missing Semester of Your CS Education for things like Git, text editors, and the terminal that students are often left to figure out as they go. When a friend is sitting next to you, this usually takes about two seconds. They lean over, point, and say, “Right there.” You click it yourself. The next time you open the app, you remember where it was. That’s where ShowMe came from. We wanted that same little bit of help when we’re figuring something out on our own. You tell it what you’re trying to do, and a second cursor points to the spot on your screen. You take it from there.
What it does
When you get stuck in an app, you can tell ShowMe what you’re trying to do. A second cursor points to where you need to click. Once you click, ShowMe looks at the screen again and guides you through the next step.
How we built it
Mac client: Python, PyQt6, PyObjC
We applied a click-through overlay. A transparent, full-screen window above every app draws a breathing glow around the next control. It ignores the mouse, so you click straight through it.
We also incorporated a real-time voice with ElevenLabs Agents. You talk to ShowMe like a phone call and can interrupt it mid-sentence. The agent drives the app through two client-side tools:
User Workflow:
show_next_step: take a screenshot → ask Gemini for the next step → draw the box → return the instruction → look_at_screen: answer a question about what's on screen →next_guess, a prediction of the following step and the text on its button. The moment your click changes the screen, on-device OCR (macOS Vision) finds that text and draws the box right away. Gemini checks it in the background and corrects it if needed. This saves tokens and time.
Challenges we ran into
ShowMe kept interrupting itself: On laptop speakers, the microphone heard ShowMe's own voice, and the agent cut itself off after two words. Echo cancellation lowered the echo by about 14 dB, and the leftover echo never passed the speech detector.
The room was full of other people talking: In our noisy hackhall, the speech detector fired on 76% of frames with nobody talking to ShowMe. Detecting speech wasn't enough; we needed to know who was speaking. Our first speaker model (CAM++) couldn't separate voices, so we switched to WeSpeaker. Enrolling in a quiet spot raised our own voiceprint scores from a median of 0.57 to 0.72.
Speed: Waiting for the model after every click used to take 4–7 s. Lower thinking effort cut a step to about 2 s, and next_guess + local OCR now shows the next box about 80 ms after the screen settles.
Small menus didn't count as changes: A dropdown barely moves the average pixel difference, so we count the fraction of pixels that clearly changed.
Accomplishments that we're proud of
91% box accuracy on our screenshot evaluation set, 6/6 on the VS Code clone steps, and 13/13 on chat-action tests. A full VS Code "clone a repo" runs in 4 steps, with every highlight on the right control. It works on any app. When a link opened Chrome mid-task, ShowMe noticed and asked the user to switch back. A voice conversation you can interrupt, on open laptop speakers, in a noisy room, with only our own voices allowed to interrupt. No API keys on the client: Everything sensitive stays on the backend behind signed URLs and an access token.
What we learned
Vision models are good at deciding what to click. The hard part is timing and state: when to look, when to wait, and when the user is just talking. Guess, then verify: A fast local guess plus a slower model check feels instant without giving up accuracy. Measure, don't guess: Our first voice threshold (0.5) would have let 28% of bystanders interrupt. Recording real numbers in the real room changed our design. Asking for one step at a time: with a checkable "what should happen next," makes the whole loop self-correcting.
What's next for ShowMe
Saying what to type, not just where, for input fields. Adaptive voiceprints: that keep learning your voice during a session. Windows and multi-monitor support. Shareable walkthroughs: record one successful run and replay it for a whole class or team.
Built With
- claude
- elevenlabs
- elevenlabs-agents
- gemini-api
- google-gemini
- python
