ASHA is learning to see before she clicks
After submitting the first demonstrator, I continued testing ASHA on my real Windows desktop. The most important lesson was simple: understanding what I mean is not enough. ASHA must also verify where the target really is before acting.
This update improves the complete desktop-control path:
- ASHA now distinguishes between opening an application and interacting with something inside it. “Open Inbox” should no longer be mistaken for launching an application named Inbox.
- Screenshot coordinates, desktop coordinates, OCR results and Windows UI Automation bounds are translated into one consistent coordinate space.
- ASHA can use semantic information such as visible names, roles and parent controls instead of relying only on a model’s estimated pixel coordinates.
- Tools are revealed progressively, only when they are needed. This reduces model context and prevents the model from calling tools that are not available in the current phase.
- New telemetry records which capability ASHA selected, which tools were available, how a target was grounded and whether an action actually reached the desktop.
- Desktop actions remain permission-gated and visible to the human.
Nothing is hardcoded specifically for Outlook, Inbox or GMX. Those are simply useful real-world tests for the general desktop-control system.
The current repair passes 54 desktop and runtime tests plus 9 shared-attention tests.
ASHA is still learning, but the direction is becoming clearer: she should not confidently claim that something happened. She should look, ground, act and verify—with the human visibly in the loop.
Log in or sign up for Devpost to join the conversation.