Inspiration

Learning unfamiliar desktop software is frustrating when instructions refer to menus that look different from the interface in front of you. Screen Guide Assistant helps people find the next step in their actual window while remaining in control.

What it does

Screen Guide Assistant is a Windows desktop application that responds to a typed or spoken question by highlighting a relevant control or estimated screen location. After the user clicks or hovers to open a menu, it can read the updated interface and guide the next step. The transparent overlay lets clicks pass through; the user performs the actions.

How we built it

The application uses Windows UI Automation to read available controls and verify their identity and boundaries before highlighting them. It can also capture the visible target window and use a vision-capable model to estimate locations where controls are not exposed. Users can choose controls-only, image, hybrid, or fallback modes. It supports configurable model services and a Codex plugin that uses the current conversation model. Speech recognition can run locally.

Challenges and what we learned

Desktop applications expose interface elements differently, and menus change as people interact with them. We combine control metadata with visual context and clear stale highlights when the window changes. Image-based positions are estimates and may be wrong. Real-world accuracy depends on the application and model.