Inspiration

Every day we reach for the mouse, and every day it ties our hand to the desk. We noticed that most of us already wear a powerful motion sensor on our wrist: the Apple Watch. It knows exactly how our arm moves, but it can't control the Mac in front of us. So we asked a simple question: what if your watch could be your mouse?

We also thought about people for whom a mouse is hard to use, like anyone with a sore or injured hand, and presenters who want to walk around while controlling their slides.

What it does

Flare turns your Apple Watch into a mouse for your Mac.

  • Point: lift your arm toward the screen and the pointer follows. Lower your arm to the keyboard and it lets go.
  • Tap: touch your thumb and finger together to click.
  • Flick: a quick twist of the wrist is a right-click. Twist the other way to grab and drag.
  • Scroll: turn the Digital Crown.
  • Trackpad: drag a finger on the watch screen for fine control.
  • Feel it: the watch taps your wrist for every click, so you don't need to look at it.

Flare also includes:

  • The Dojo: a 3D training space where you learn the gestures, play practice games, and calibrate Flare to your own arm.
  • June: a voice assistant powered by Google Gemini. Say "Hey June" and ask about anything on your screen. She can summarize a page, explain what you're pointing at, search the web, type for you, and work with your files, and she answers out loud.

How we built it

  • Watch app (Swift and SwiftUI): reads the watch's motion sensors 100 times a second, plus the Digital Crown and the touch screen. It runs a workout session so it keeps streaming when your wrist drops and the screen dims.
  • Connection (Bluetooth LE): the watch talks straight to the Mac, without the iPhone in between, sending about 50 updates a second.
  • Security: the first time you connect, both screens show the same 6-digit code, and only a paired watch is accepted after that.
  • Motion engine (Mac): turns raw sensor data into smooth pointer movement. It uses gravity to work out which way is "up", so twisting your wrist doesn't move the pointer, and it filters out hand shake.
  • Gesture detection: a finger tap shows up as a short, sharp spike in the watch's accelerometer, and a flick is a fast twist that springs back.
  • Mac app (SwiftUI): a native macOS app that moves the real cursor using standard Mac mouse events.
  • The Dojo (SceneKit): a 3D scene with an animated arm and watch that copies your real movements.
  • June: on-device speech recognition for "Hey June", with Google Gemini for the answers and spoken replies.
  • Testing: we built a simulated watch so we could test the motion engine and the connection automatically, with over 200 checks, before trying it on a real wrist.

We also used AI coding tools, including Claude Code, to help write and debug the code, and tested everything on a real Apple Watch.

Challenges we ran into

  • Which way is the watch worn? Left wrist, right wrist, and which side the crown is on all change what "up" means. We solved it with a calibration step in the Dojo.
  • Telling a tap from normal movement: moving your arm also shakes the sensors. We had to pick out a tap's short, sharp spike from everything else, and let each person tune it to their own hand.
  • Keeping the watch awake: the watch normally pauses apps when you lower your wrist, so we used a workout session to keep the data flowing.
  • A tricky crash in June: June sometimes crashed or froze the app. The cause was June asking macOS about Flare's own window from a background thread, which quietly damaged the app. Finding it took crash reports, thread snapshots and a lot of testing.
  • Mac permissions: controlling the cursor, Bluetooth and the microphone each need their own permission, and getting them all working smoothly took real effort.

Accomplishments that we're proud of

  • It really works: you can point, click, scroll and drag on a Mac with nothing in your hand.
  • The watch connects to the Mac directly and securely.
  • The 3D Dojo makes learning the gestures feel like a game.
  • June adds a voice assistant on top of the mouse, so your watch can both point at something and ask about it.

What we learned

  • How to read and clean up real motion sensor data.
  • How Bluetooth Low Energy works between two Apple devices.
  • How much careful design it takes to make a new way of using a computer feel natural.
  • How to track down hard bugs using crash reports and thread snapshots.

What's next for Flare

  • Measure the delay and battery use, and improve both.
  • Encrypt the connection, not just authenticate it.
  • Smarter gesture detection that learns from each user.
  • Bring Flare to more devices and make it a real accessibility tool.

Built With

  • accessibility-api
  • avfoundation
  • bluetooth
  • corebluetooth
  • coremotion
  • cryptokit
  • google-gemini-api
  • healthkit
  • macos
  • scenekit
  • speech-framework
  • swift
  • swift-package-manager
  • swiftui
  • watchos
  • xcode
Share this project:

Updates

Submission history