souris

souri interface

Inspiration

i've always felt like there are a lot of times where you could use an extra input device, whether that's a mouse instead of trackpad to play league of legends or being unable to use your keyboard cause you're eating something with one of your hands. so, i wanted to make something that you could still use in this case to work without restricting your workflow at all (and potentially even helping you do better - it's like the opposite of vim but for the same goal).

What it does

it allows you to control everything on your computer from just using your phone (never need a mouse or keyboard or other input devices again). instead, you can click into things, quick open actions, dictate and delete, etc.

How we built it

two native swift apps, using only apple's frameworks (no third-party packages):

the iphone app does all the work. the rear camera looks down at the desk and arkit's world tracking works out where the phone is. arkit only updates 60 times a second and runs about 37 ms late, so the gyroscope handles swivels and tilts instantly, and the accelerometer, through a kalman filter we wrote, predicts slides before arkit catches up. a 1€ filter smooths out jitter and a speed-based curve adds pointer acceleration.
the mac app is tiny. it's a menu-bar app that receives the phone's packets and turns them into real mouse and keyboard events. it paces cursor moves to the display's refresh rate so motion looks smooth.
three links. usb goes through usbmuxd, the same channel xcode uses to talk to your phone, and has a p95 delay of 3.5 ms. bluetooth le has a pacer we built that adapts to the link. wi-fi is the fallback. packets carry running totals instead of deltas, so a lost packet never loses motion.
quick cast actions + dictation. all thanks to using the mac's actual existing functionality and pressing the keys on phone press.

Challenges we ran into

venue wi-fi. cal hacks wi-fi isolates devices (the phone and mac couldn't even ping each other), peer-to-peer wi-fi delivered packets in 300–500 ms clumps, and hotspot was banned. so we built a usb link and a bluetooth link.
bluetooth lag. the first version queued updates behind each other, so the cursor ran 80–170 ms late in bursts. the fix was a pacer that sends one update at a time and backs off when ping times show a queue building.
ios locks down the buttons. apps get no button events while the camera is running, so we read the volume buttons by watching the phone's volume and resetting it after every press. the side button can't be used at all.
mac quirks. a shortcut we pressed left control held down, so every left click after it was a right click. dictation drifted out of sync until the mac app followed whether it was really recording.

Accomplishments that we're proud of

no extra hardware and no dependencies: just an iphone, a mac and apple's frameworks.
everything you'd do with a mouse and keyboard from one phone: point, click, drag, scroll, type, enter, delete and shortcuts.

What we learned

arkit is great at knowing where a phone is but too slow on its own for a cursor. fusing the gyroscope and accelerometer with it is what makes it feel instant.
latency is about the stalls, not the average. venue wi-fi had a fine average and still froze for half a second at a time.

What's next for souris

smarter dictation with apple's new speech api: better pickup from further away, and custom words so names like "ghostty" come out right.
per-app profiles, so the buttons can do different things in a game than in a browser.
making a left handed grip + getting it onto android so anyone can try it.