Project Update: ASHA Can Now Act With a Virtual Cursor
Since the last update, ASHA’s computer-control layer has become much more complete.
ASHA now connects directly to the CUA desktop driver. Her virtual cursor can move, click, double-click, right-click, drag, and scroll without taking over the user’s physical mouse whenever background interaction is supported.
The virtual and physical mouse are now separate, explicit capabilities. ASHA knows which one is enabled and which executor is actually available. If a background action cannot be performed safely, she does not silently move the real mouse or pretend that the action succeeded. She stops and asks before using physical input.
I also improved the temporary permission system. When computer-control settings change during an active control session, ASHA now shows a Restart computer control button. This activates the new permissions without requiring a new conversation or shared-attention session.
Visual grounding has also become more reliable:
- Questions such as “What do you see?” now require a fresh desktop observation.
- If the interface changes between observation and action, ASHA rejects the old coordinates.
- She automatically captures one fresh view, grounds the unfinished request again, and retries safely.
- Short follow-ups such as “Inspect it” continue the existing desktop task instead of losing its context.
- CUA timeouts are treated as uncertain results, preventing accidental duplicate clicks.
Nothing is hardcoded for Outlook, LM Studio, my computer, or a particular workflow. The same capability and grounding architecture is intended to work across different applications, computers, and users.
The current build passes:
- 84 ASHA interaction, perception, permission, and safety tests
- 10 visual-cue and session tests
- Release builds with zero warnings and zero errors
The result is beginning to feel much closer to the original idea: a desktop companion that can see, explain, demonstrate, and act while keeping the human visibly in control.
Log in or sign up for Devpost to join the conversation.