-
-
AirPlay eyes, USB HID hands: the app runs on the same iPhone. No computer at runtime.
-
Nemotron on Nebius: 82.6% action accuracy; 3.31 s median response. Recorded-screen actor evaluation, 69 calls.
-
Same-harness model comparison: action quality and client-observed response time on 23 iPhone screens.
-
Real iPhone beside the app mirror, followed by visual pointer calibration on the same device.
Give AI eyes and hands on a real iPhone
Handsy lets a vision model operate a real iPhone through a $7 ESP32-S3. It sees the screen through AirPlay and acts through USB mouse and keyboard input. The native iPhone app runs the agent loop, so the app workflow needs no computer at runtime and no jailbreak.
The open-source project has 3.8k GitHub stars and 501 forks as of October 25, 2026. These are community interest signals, rather than active-user counts.
Inspiration
An agent can reason about a task and still be unable to do it on the phone in your hand. Many useful workflows live inside mobile apps without an automation API. Developers and testers need a way to connect model decisions to real iPhone input without maintaining a device farm.
Handsy makes that bridge tangible: a small, phone-powered board supplies the eyes and hands, while the user chooses the model that makes decisions.
What it does
- Turns a natural-language goal into taps, swipes, typing and navigation across apps.
- Runs the planner and actor in a SwiftUI app on the same iPhone being controlled.
- Saves useful tasks for Siri, Shortcuts, widgets and Control Center.
- Shows progress and a stop control through Live Activities and the Dynamic Island.
- Records runs for step-by-step replay with action overlays and trace export.
- Offers a separate CLI path for computer-based agents to operate the same hardware.
The 2:54 competition video combines the English project introduction, a 68-second real hardware CLI demonstration, and the NVIDIA-on-Nebius evaluation results. The app introduction uses a local Qwen model; the CLI segment is driven by Claude Code and is edited and partly sped up. The Nebius evaluation uses the app's actor prompt and is described below.
How we built it
Eyes: the ESP32-S3 creates a Wi-Fi hotspot and receives the iPhone's AirPlay H.264 stream. It passes compressed frames over BLE to the app on that same phone. VideoToolbox decodes them; the board does not need to decode video.
Brain: the app sends screenshots to an OpenAI-compatible vision endpoint. A planner maintains the goal and an actor returns the next structured action. We evaluated NVIDIA Nemotron 3.5 Super VL through Nebius Token Factory, using the actual app actor prompt and recorded real-iPhone screens.
Hands: the app sends commands over BLE. The board presents itself to iOS as a USB HID mouse and keyboard. AssistiveTouch converts pointer clicks into taps and drags into swipes.
Feedback: relative pointer movement needs calibration. Handsy tracks and re-homes the pointer, checks its position visually before precise taps, waits for the screen to settle and examines the result. Connection loss pauses the run for recovery. The phone reaches cloud inference over cellular data while Wi-Fi is connected to the board.
NVIDIA on Nebius: measured actor performance
On October 18, 2026, we made 138 scored inference calls to nvidia/Nemotron-3.5-Super-VL through the Nebius Token Factory OpenAI-compatible API. Both image presets returned 69/69 successful, parseable responses with zero retries.
| Image preset | Action accuracy | Median response | p95 response |
|---|---|---|---|
| 540 px wide, JPEG quality 60 | 73.9% (51/69) | 3.00 s | 3.69 s |
| 718 px wide, JPEG quality 75 | 82.6% (57/69) | 3.31 s | 3.98 s |
Method: single-step actor evaluation on 23 recorded iPhone screens from two tasks, three repetitions per screen and preset, one warm-up excluded per preset. Action accuracy uses deterministic checks of target coordinates and action parameters, including text, keys, swipe direction, loops and premature completion. Full task completion was not measured. Repeated screens are correlated; latency includes the client network path and serving.
Larger images improved Nemotron's action accuracy in this corpus. The model consistently produced parseable actions, but parsing alone did not guarantee correct behavior. Other models scored higher: at the 718 px preset, Qwen3.8 Flash on Bailian achieved 97.1% at 1.59 s median, while our local Qwen3.8 Flash Next on two DGX Sparks achieved 91.3% at 6.62 s. These results help us choose models by measured tradeoffs.
Challenges and what we learned
The hard engineering is closing the control loop: low-bandwidth BLE video, Wi-Fi/Bluetooth coexistence, relative mouse coordinates, background execution and recovering from a board or mirroring disconnect. A model's action schema can be perfect while its chosen action is wrong. Run replay and parameter-aware evaluation exposed loops and premature completion that a simple parser check would miss.
Nebius's OpenAI-compatible endpoint let us reuse the actor harness without hosting another model server. In the default small-image sample, its Nemotron route had a tighter p95 than the NVIDIA API route (3.69 s versus 8.92 s), while the NVIDIA route had a lower median (1.99 s versus 3.00 s). This is an observation from our client path; it does not establish a general provider ranking.
What we are proud of
We moved the main workflow from a computer-side prototype to a phone-and-board system, implemented the firmware and native app, made runs inspectable, and published the source and setup instructions. The project's October development falls within this hackathon's submission period.
What's next
Improve action quality and recovery on more apps, expand the evaluation corpus, measure full tasks with live hardware, support landscape orientation and non-ASCII input, and reduce time per step through better action batching.
Current limits: portrait orientation and ASCII input; BLE constrains mirror resolution and frame rate; locking the phone stops mirroring; protected screens such as passcode entry and Apple Pay are not visible. App Store distribution is still pending. The project is intended for supervised use on authorized devices.
Try it and inspect the work
- Source code and setup
- Website and documentation
- Real hardware CLI demonstration
- Short app demonstration
The app is primarily Swift/SwiftUI, the board firmware is C/C++ on Arduino ESP32 with NimBLE and TinyUSB, and the evaluation harness is Python. The main project is MIT licensed; the UxPlay-derived AirPlay code and linked firmware have GPL-3.0 licensing requirements, documented in the repository.
Log in or sign up for Devpost to join the conversation.