TransitLens

Inspiration

Our inspiration came from trying to get around Gainesville by bus. In our experience, Apple Maps did not provide usable local bus routing, and the apps that showed live buses still made it difficult to figure out which vehicle to catch. Seeing dots on a map left us piecing together routes, destinations, and travel directions ourselves.

Even when we knew a bus was nearby, we still had to connect its marker on a screen with the vehicle coming down the street. We wanted that connection to feel immediate.

What if you could point your phone down the street, see nearby buses labeled by route, and ask, “Does this bus go to Shands?”

That became TransitLens: look down the street, find your bus. We built a transit heads-up display for Gainesville RTS that combines geographic bus markers, an interactive map, and a Gemini voice copilot. It works as a handheld phone experience and includes a stereo mode for Google Cardboard.

What it does

TransitLens connects the rider’s surroundings with the transit network.

Find buses by direction. GPS, device orientation, and reported vehicle locations position route markers at their geographic bearings.
Look to select. Holding your gaze on a marker selects a bus and reveals its destination and available vehicle information.
Ask about what you see. Say “Hey Gemini, where is this bus going?” to ask about the focused vehicle.
Explore direct trips. Ask for routes to a destination, choose an option, and see approximate direction and distance to its boarding stop.
Keep the map within reach. A persistent minimap shows streets, routes, and stops, with an expanded view for exploration.
Read a sign on request. An explicit bus-sign request sends a single camera still to Gemini for interpretation.
Share the experience. WebRTC casts the camera and composited HUD to a laptop, where the viewer can record it.

A separate showcase mode animates ten synthetic buses around UF using captured RTS route geometry. Demo vehicles and scripted arrival times are explicitly labeled.

How we built it

We split the project into a TypeScript web client and two Python services. The bus service normalizes RTS vehicles, route patterns, stops, and available arrival predictions. The voice service interprets requests with Gemini, retrieves transit information, and returns answers alongside structured interface actions.

The client keeps the HUD, map, selected vehicle, and boarding guidance connected through shared state. Selecting a route by voice updates the same experience used by touch and gaze.

The math behind the HUD

The geographic overlay uses sensors and geometry. It does not depend on visually recognizing buses in camera frames.

For a rider at latitude and longitude pair 1 and a bus at pair 2, we calculate the initial geographic bearing:

$$ \theta=\operatorname{atan2}\left( \sin(\Delta\lambda)\cos\varphi_2,\, \cos\varphi_1\sin\varphi_2- \sin\varphi_1\cos\varphi_2\cos(\Delta\lambda) \right) $$

Here, phi represents latitude and lambda represents longitude. After converting the bearing to degrees, we calculate its signed offset from the camera heading:

$$ \delta=((\theta-h+540)\bmod360)-180 $$

That offset becomes a horizontal screen position through a pinhole-camera projection:

$$ x=\frac{W}{2}\left(1+\frac{\tan\delta}{\tan(F/2)}\right) $$

W is the view width, and F is the effective horizontal field of view. Our implementation adjusts that field of view for camera zoom, portrait orientation, and the cropping introduced when the video fills the screen. Those corrections matter because the displayed camera image can show a different angular range from the original stream.

We also use the Haversine formula for great-circle distances between riders, buses, and stops.

Finding where the camera actually points

A phone’s raw compass value does not always describe the rear camera’s direction, especially when the phone is sideways inside Cardboard.

We convert device orientation into a rotation matrix and transform the camera’s forward vector into East-North-Up coordinates:

$$ \mathbf{c}=-R\hat{\mathbf{z}}, \qquad h=\operatorname{atan2}(c_E,c_N) $$

We smooth the direction vector before extracting heading, which avoids jumps when crossing north. The system also applies location- and date-based magnetic declination when available, with separate handling for Safari’s compass behavior.

Grounding the assistant in transit data

Gemini interprets requests such as “show routes to Shands” or “take the first option.” Deterministic backend code resolves transit queries, and the client validates structured actions before applying them.

This lets conversational input control real interface behavior while keeping route geometry, vehicle identity, and available predictions tied to transit data. Ambiguous destinations or vehicle choices trigger clarification.

Ordinary questions use transit context. Camera images are sent only for explicit sign-reading requests, and those images are excluded from conversation history.

Understanding progress along a route

A nearby bus may be traveling away from a stop or following a different route pattern. Straight-line distance alone cannot establish that it is approaching.

We project vehicle positions onto route segments and calculate progress along the route. For a segment from a to b and vehicle position p, the local planar projection is:

$$ t=\operatorname{clamp}\left( \frac{(p-a)\cdot(b-a)}{|b-a|^2},0,1 \right) $$

The implementation scales longitude by latitude and uses cumulative geographic segment lengths to estimate remaining route distance. It rejects distant matches, conflicting travel directions, and ambiguous loop positions. Arrival times come from available stop-specific predictions rather than being inferred from distance.

Our stack

Frontend: TypeScript, Vite, HTML/CSS, Canvas 2D, and MapLibre GL JS.
Backend: Python, FastAPI, Uvicorn, HTTPX, and Pydantic.
Transit data: Gainesville RTS BusTime vehicles, routes, patterns, stops, and predictions.
AI and speech: Google Gemini through the Google Gen AI SDK, Gemini-generated speech, browser speech recognition, and browser speech synthesis as a fallback.
Sensors and geometry: Geolocation, device orientation, quaternion and rotation-matrix calculations, and PyGeoMag.
Casting: WebRTC and canvas compositing.
Android shell: Capacitor with Java bridges for native compass, camera field of view, and screen information.
Deployment: Docker Compose and Render configuration.

Challenges we ran into

Getting the overlay to match the camera view. Camera cropping, phone rotation, magnetic north, and headset placement all affect alignment. We implemented orientation-aware projection and calibration controls for zoom, field of view, tilt, lens spacing, horizontal shift, and image size.

Making voice interaction tolerate natural pauses. Browser speech recognition can mark a phrase complete before someone finishes their question. We collect speech until approximately 1.3 seconds of silence and support a fresh wake phrase to interrupt playback. Cancellation handling prevents an older response from taking over a newer interaction.

Working within API limits and outages. Vehicles and route catalogs change at different rates, so we cache them separately and budget upstream requests. When refreshes fail, available cached data retains freshness information. Live mode never silently substitutes synthetic vehicles.

Keeping uncertainty visible. A published route, a reported vehicle, and an arrival prediction answer different questions. We preserve those distinctions throughout the backend, assistant, and interface. Missing information remains unknown.

Accomplishments that we're proud of

We connected gaze, voice, maps, and live transit data in one shared experience. A rider can focus on a bus, ask about it, explore a destination, and display boarding guidance without repeatedly reselecting context.

We are also proud of the implementation details behind that experience: conservative route matching, validated assistant actions, explicit demo labeling, speech fallbacks, and casting that lets other people see inside the headset.

What we learned

The hardest part of an AR interface is making its coordinate systems agree. A small error in heading or field of view can make an otherwise correct geographic marker feel misplaced.

We also learned that conversational transit tools need explicit uncertainty. Route coverage does not confirm that a bus is operating, and distance does not establish an arrival time.

Finally, designing for a headset changes interaction priorities. Gaze selection, readable labels, short spoken answers, and adjustable calibration become central to usability.

What's next for TransitLens

Our next step is physical testing across phones, outdoor locations, microphones, and Cardboard headsets to evaluate alignment and usability.

We also want to add transfer planning, verified pedestrian directions to boarding stops, and stronger support for multiple simultaneous casting sessions.

Today, trip discovery supports direct routes, and boarding guidance provides approximate straight-line direction and distance. Building on that foundation will help TransitLens answer more of the questions riders face between finding a route and boarding the right bus.

github: https://github.com/acheng2007/transitlens