Project Pigeon

Inspiration

The world is built for humans. Signs, labels, and layouts assume a person is reading them. Robots, AR characters, and other digital consumers cannot parse a room the way we can. We wanted to close that gap. If a character needs to walk from the front door to a specific seat, it has no map that understands what a "door" or a "seat" even is. Project Pigion gives any space a semantic layer that machines can read and act on.

What it does

Scan the world with any device. Our AI pipeline recognizes objects and text in the scan, then builds a semantically understandable map of the space. You can query the world in natural language ("where is the nearest exit," "find an open seat") and modify it the same way. The result is a spatial graph that both humans and non-human agents can navigate.

How we built it

We capture the environment through device-based scanning and reconstruct it into a 3D map. Early versions used Apple's spatial capture APIs to generate the base geometry. We later integrated xgrid to qualify for a category, which meant merging xgrid's representation onto the Apple-generated map. Aligning the two systems is the core of the pipeline, since each describes the same physical space in its own coordinate frame and its own vocabulary. On top of the merged scan, an object and text recognition pipeline labels each element and links it into a queryable semantic graph. A natural language interface sits on top so users can ask questions and issue edits. We also have Pigeon Buddy that show you the way.

Challenges we ran into

Raw scans give geometry without meaning, so turning point clouds and surfaces into labeled, queryable objects was hard. Merging xgrid onto Apple's map proved especially finicky, because the two systems label the same world differently and their representations do not line up cleanly. Reconciling those descriptions so a single object was not counted twice or lost took real effort. Text in the real world is small, angled, and inconsistent, which stressed our recognition step. We also had to swap our mapping backend partway through, since we adopted xgrid after starting on a different foundation. Keeping the semantic graph consistent across devices added further complexity.

Accomplishments that we're proud of

We built a working pipeline that takes an ordinary scan and returns a machine-readable model of a space. Natural language querying and modification work end to end. We proved that a character can reason about a path from a door to a seat using labels our system generated.

What we learned

Geometry alone is not navigation. The hard part is the semantic layer that tells an agent what each surface and sign means. We learned how to combine scanning, object recognition, and text recognition into one coherent representation. We also learned how much work it takes to align two mapping systems that speak different languages about the same space. We also learned to stay flexible on infrastructure, since acquiring xgrid mid-project reshaped our approach.

What's next for Project Pigion

We want broader device support so any phone or headset can contribute scans. A central goal is robust cross-system matching, so maps from xgrid, Apple, and future sources fuse into one consistent representation despite their different coordinate frames and labels. Next we plan to expand the object and text vocabulary, improve real-time updates as spaces change, and open the semantic map to third-party agents and robots. The long-term goal is a shared, machine-readable layer over the physical world.

Built With

• apple
• tripo
• xgrid