ASHA is learning to see before she clicks
After submitting the first demonstrator, I continued testing ASHA on my real Windows desktop. The most important lesson was simple: understanding what I mean is not enough. ASHA must also verify where the target really is before acting.
This update improves the complete desktop-control path:
- ASHA now distinguishes between opening an application and interacting with something inside it. “Open Inbox” should no longer be mistaken for launching an application named Inbox.
- Screenshot coordinates, desktop coordinates, OCR results and Windows UI Automation bounds are translated into one consistent coordinate space.
- ASHA can use semantic information such as visible names, roles and parent controls instead of relying only on a model’s estimated pixel coordinates.
- Tools are revealed progressively, only when they are needed. This reduces model context and prevents the model from calling tools that are not available in the current phase.
- New telemetry records which capability ASHA selected, which tools were available, how a target was grounded and whether an action actually reached the desktop.
- Desktop actions remain permission-gated and visible to the human.
Nothing is hardcoded specifically for Outlook, Inbox or GMX. Those are simply useful real-world tests for the general desktop-control system.
The current repair passes 54 desktop and runtime tests plus 9 shared-attention tests.
ASHA is still learning, but the direction is becoming clearer: she should not confidently claim that something happened. She should look, ground, act and verify—with the human visibly in the loop.