Inspiration

Vexo remembers what you said. We wanted it to remember **where* you said it.*

Most of the places we get lost in have no map: a hospital floor, an office, a parking garage, a friend's house. GPS doesn't work indoors, and nobody has drawn a floor plan for your route. We wanted a wearable that builds that map from the way you walk, so you can say "this is the entrance" and later "take me back to the entrance."

We also wanted it to work for people who can't rely on one sense. If you can't see, it speaks to you. If you can't hear, it shows light and buzzes. If you can't speak, you can tap and type. Whichever sense you're missing, you can still find your way back.

What it does

  • Track where you walk. The iPhone's camera and motion sensors (ARKit) track your path while the camera can see, at any phone angle. If the camera is covered or the phone is in a pocket, tracking continues on the Vexo band's motion and step sensors, fused with the iPhone's IMU, pedometer and barometer. No floor plan, no GPS, no laptop.
  • Save places by voice. Double-tap the band and say "this is the kitchen". The name is pinned to your current point on the route.
  • Guide you back. Say "take me to the kitchen". Wayback Trail walks your saved path backwards, leg by leg, gives one direction at a time, warns you before turns, and resets your position when you arrive, which cancels drift.
  • Turn you the right way with buzzes. The band has no vibration motor, so every buzz comes from the phone. One rule: a steady pulse while you face the right way, silence when you don't. Turn until the pulse comes back, then walk. Directions are spoken through the phone's speaker or AirPods. Arrival is 3 long buzzes.
  • Show light on the wrist. Green means the route started, amber means recording, purple means a place was saved, white means guiding, red means lost.
  • Re-anchor. Stand at a saved place and say "I'm at the stairs" to snap your position back.

How we built it

Camera + band → path (iPhone app). While the camera can see, ARKit's visual-inertial tracking (camera + gyro + accelerometer) gives the position and facing, with the phone held at any angle. While it tracks, the band's step counter and the phone's footfalls are compared with the distance the camera measures, which learns your stride live. When the camera is covered or the phone is pocketed, tracking falls back to the band and the phone's motion sensors below, continuing from the same position.

Band → path (fallback). The band streams motion samples at 50 per second and step counts. Heading comes from the gyro projected onto gravity, where gravity $\hat{g}$ is a low-pass filter of the accelerometer:

$$\omega_{\text{yaw}} = \vec{\omega} \cdot \hat{g} - b, \qquad \theta(t) = \int \omega_{\text{yaw}}\, dt$$

Here $b$ is the gyro bias, measured while you stand still. Each step moves you by the step length $L$ in the direction you face:

$$x \mathrel{+}= L \sin\theta, \qquad y \mathrel{+}= L \cos\theta$$

The heading is smoothed with a rolling median and held on a $90^\circ$ grid, which keeps hallways straight.

Voice. A double-tap on the band turns its microphone on. Deepgram turns speech into text, Gemini 4 (gemini-4-hydrogen) picks one of the app's tools with native function calling, and ElevenLabs answers out loud in the voice "River", with MiniMax as the backup voice. One rule held throughout: the model never computes a position. Gemini picks the action and matches place names, and our code computes every coordinate, distance and turn.

Guidance. The route is a graph of the path you actually walked. Guidance cleans small wobbles out of the walked line, merges legs that point the same way and turns them into spoken directions like "Continue straight for 12 steps" or "Turn left now, then walk 13 steps."

iPhone app. A native SwiftUI app plays the buzzes with Core Haptics, because web pages can't vibrate on an iPhone. It runs with no laptop: it connects to the band directly over Bluetooth (the kit only supports laptops, so we implemented the band's protocol in Swift) and runs tracking, guidance and voice on the phone. One screen: the map of your walk, Start/Stop, Talk, Save place, and saved places you tap to be guided back.

Web pages. A trail page shows the live path, a facing cone, the band's connection and battery, and the voice controls. There's also a team map and a phone page that vibrates on Android. No API key ever reaches a page: keys are read only in Python, and only under our own VEXO_ names.

Testing. 115 unit tests. Every real walk is logged to a CSV (the iPhone app logs every sensor reading too), and a replay tool runs any log through the tracker, so we tuned the tracker on recorded walks instead of guesses.

Challenges we ran into

  • The band can't do what a phone does. It has no compass, no vibration motor, no speaker and no barometer. Heading had to come from the gyro alone, the buzzes moved to the phone, and spoken guidance comes from the laptop or the iPhone.
  • A wrist is not a body. Arm swing rocked the raw heading by up to about $100^\circ$, so the first turn detector never settled and one walk ended about 22 m off. A rolling-median heading on a $90^\circ$ grid fixed it.
  • Calibration caught a moving hand. A fixed calibration window measured the gyro bias while the arm moved, leaving it 1.2°/s wrong. Now calibration waits for real stillness and re-estimates the bias whenever you stop.
  • The band's step counter started about 15 s late and updated once a second, so we also built a footstep detector on the accelerometer.
  • One connection at a time. Only one program can hold the band over Bluetooth, so we built several ways to connect: the laptop directly, a USB cable, Chrome, or the iPhone.
  • The light turns solid blue while the mic is on, and we can't change it then. So we listen first, switch the mic off, and only then show the answer on the light.
  • A phone moved in the hand looks like a turn. Motion sensors alone can't tell "you turned" from "the phone moved", so position comes from the iPhone's camera when it can see, and the band's sensors carry the route when it can't.

Accomplishments we're proud of

  • On a real 120 m walk with the phone in hand (camera tracking, the Vexo band for every command, no laptop), the path ended 3.0 m from where it started.
  • On our recorded wrist walk (walk3), a 59 m loop, the replayed path ends 1.0 m from where it started.
  • It works eyes-free: double-tap, speak, and follow the light and the buzzes.
  • The iPhone talks to the Vexo band directly over Bluetooth, with no laptop in the loop.

What we learned

  • Fix the hardware limits early. Listing what the band lacks (compass, motor, speaker) shaped the whole design, from gyro-only heading to phone haptics.
  • Log every real walk. Each tracker fix came from replaying a real log, not from tuning on synthetic data.
  • Keep the model and the math apart. Letting Gemini choose what to do, while code decides where, made the system predictable and testable.
  • Accessibility means more than one channel. Voice, light, buzzes and typing each cover a different person.

What's next

  • Weight each sensor by its uncertainty (a Kalman filter) instead of switching between camera and band, so pocket tracking gets close to camera tracking.
  • Test longer and multi-floor routes (the barometer already records floor changes).
  • Merge the Pipecat voice pipeline (it's on a branch).

Wayback Trail is an experimental navigation aid, not a safety-critical wayfinding system. Routes stay on your own device, and "delete all" forgets every route.

Built With

Share this project:

Updates

Submission history