Why I built it
Software tutorials show someone else’s screen, and an agent that completes a task for you may leave you unable to repeat it. Ghost Hands aims to make computer help hands-on: watch the steps on your own Windows screen, then practice them.
What it does
Ask a question by holding F8 to speak or pressing F7 to type. The app captures the screen, uses GPT-6.1 Sol computer use to plan steps, and displays a ghost cursor that moves to each target while captions and speech explain the action. Press F9 for “Your turn”: rings mark the recorded targets and wait for your clicks. F10 replays the last run without another model call.
A code-level guard checks the accessible name at each click target and stops before actions such as Send, Pay, Delete, Submit, Buy, or Transfer. Moving the mouse or pressing Esc also stops the run. The repository documents end-to-end checks in Notepad and Windows Settings; the demo shows a Google Sheets task.
How I built it
The desktop app uses Python, PyQt6, Windows UI Automation, and the OpenAI Responses API computer-use tool. OpenAI speech-to-text handles push-to-talk input; text-to-speech narrates the steps. A separate UI Automation helper process, per-monitor DPI handling, and a capture-excluded overlay support the on-screen guide. Codex and Claude Code assisted development, as documented in the repository.
Challenges and limits
Keeping the overlay visible to the user while excluding it from model screenshots required a separate compositing approach. Ghost Hands currently targets Windows 10/11, works on the primary monitor, needs Python 3.11+ and an OpenAI API key for live AI runs, and cannot control apps running as administrator from an ordinary process. A scripted --fake mode demonstrates the overlay without an API key.
Ghost Hands was developed before any later hackathon entry; future event-specific submissions will disclose that prior work and identify any new work accurately.
Log in or sign up for Devpost to join the conversation.