GhostPoint

Ghostpoint turns iPhones into a shared sixth sense: see agents through walls, snap to them with Apple Vision, and let team supervisors watch every perspective live.

Comment

Ghostpoint

Shared spatial awareness, built into the phones you already carry.

Inspiration

In a video game, you always know where your teammates are. In the real world, one closed door and they disappear from view.

Inside an unfamiliar building, even a simple call like “I’m down the hallway” depends on everyone sharing the same mental map. We built Ghostpoint to explore a more intuitive alternative: showing participating teammates directly over your camera view, including when they move out of sight.

Our goal is to bring game-like team awareness to compatible, UWB-equipped iPhones—without external tracking hardware.

What it does

Teammate markers beyond line of sight. Ghostpoint combines Ultra-Wideband measurements with motion tracking from both phones. Once their coordinate systems are aligned, incoming ARKit poses can sustain teammate positions during temporary UWB interruptions, including when a wall or doorway obstructs the signal.

Visual refinement when teammates are visible. Apple Vision detects people in the camera feed. When a detection plausibly matches a tracked teammate’s bearing and apparent size, Ghostpoint shifts the marker toward that person. Ambiguous matches are rejected rather than guessed.

Direction at the screen edge. Arrows guide users toward tracked teammates outside the camera view, with names and distances.

HUD broadcasting. A native screen-capture pipeline is designed to share the rendered interface—not just the camera image—with a browser dashboard. The mobile client requests one of four configured broadcast slots and displays connection status. Broadcasting is video-only.

A shared team waypoint. Through the coordinator workflow, one phone receives permission to place, rename, replace, or clear a named waypoint. Placement uses detected surfaces where possible, with explicitly estimated feature-point or distance-based alternatives. Other phones display it only when their tracking and alignment can map it into the correct AR world.

Visible tracking limitations. Ghostpoint distinguishes unavailable positions, estimated tracking, camera-refined markers, and missing UWB. It also provides Bluetooth warnings, pairing diagnostics, and recovery behavior for interrupted sessions.

Ghostpoint tracks participating devices. It does not detect unknown people through walls or identify hostages.

How we built it

The interface uses React Native, Expo, and TypeScript. Custom Swift modules handle the performance-sensitive sensing, tracking, rendering, and video pipeline.

Multipeer Connectivity discovers nearby phones and exchanges Nearby Interaction discovery tokens, ARKit positions, and UWB-derived measurements. Phones can connect automatically or use a matching pairing code. Local teammate tracking does not require a cloud server.

Nearby Interaction supplies distance and, when supported and available, direction or camera-assisted world positions. RealityKit renders the spatial markers, while native UIKit labels update alongside the camera to avoid the lag of JavaScript-driven tags.

The central challenge is alignment: every phone starts with its own AR coordinate system. Our alignment layer estimates the yaw rotation and translation between those gravity-aligned worlds using paired UWB measurements and exchanged positions.

A weighted sliding-window fit emphasizes recent observations, rejects outliers, and gates measurements that contradict an established alignment. Median filtering, adaptive smoothing, and short-horizon motion prediction reduce jitter and visible lag.

Once aligned, Ghostpoint can follow a teammate’s streamed ARKit position instead of relying on each new radio reading. That fallback is bounded: it requires fresh peer poses and expires after 30 seconds without an accepted alignment update. Detected AR-world changes reset the alignment.

For broadcasting, ReplayKit supplies rendered screen frames directly to native WebRTC, without sending video buffers through JavaScript. The publisher uses WHIP signaling and stops when the app enters the background.

Challenges we ran into

Reflected radio measurements. An obstructed UWB signal can produce a misleading position rather than simply disappear. We added consistency checks so contradictory readings do not immediately pull an established alignment off course.

Labels trailing the camera. JavaScript-driven tags could visibly lag during camera movement. Moving on-screen labels into the native view kept their positioning tied to the rendering loop.

Matching people without claiming identification. A person detection is not proof of identity. We constrain visual matches using tracked position, apparent size, and ambiguity checks.

Sharing meaningful coordinates. A waypoint is only useful if another phone can interpret its reference frame. Ghostpoint carries the originating peer and AR-world epoch with the point and withholds it when mapping is unavailable.

Handling interruptions honestly. Bluetooth changes, dropped links, suspended UWB sessions, and degraded AR tracking all require different recovery paths—not a frozen marker pretending everything is fine.

Accomplishments we’re proud of

  • Implementing UWB, exchanged ARKit poses, and visual refinement in one tracking pipeline.
  • Keeping spatial markers and labels native while retaining a React Native interface.
  • Connecting HUD broadcasting with a permission-controlled shared-waypoint workflow.
  • Making degraded and unavailable tracking visible instead of hiding it.

What we learned

Spatial awareness is not just a positioning problem—it is a confidence problem.

Radio measurements, device motion, and visual detections provide different kinds of evidence. The important work is deciding when they agree, when they are unreliable, and when the interface should stop showing a position.

What’s next

We plan to validate alignment, obstructed tracking, visual matching, full HUD capture, and shared waypoints across more physical devices and environments. We also want to measure accuracy, latency, battery use, and thermal behavior, add TURN support for restrictive networks, and replace the temporary hosting address. We also plan on expanding the amount of tracking solutions to improve compatibility with a wider range of hardware, or source and produce dedicated hardware for precise, long-range, and reliable through-wall tracking.

Ghostpoint is a consumer-hardware prototype, not a validated tactical system. We are building toward shared spatial awareness that is intuitive—and honest about its limits.

Built With

Share this project:

Updates

Submission history