Inspiration
Curators hang by feel, then learn on paper that a frame is too high for a seated visitor or a print is over-lit. By then the nail holes are in the wall.
What it does
Curators pinch, drag and review a gallery in mixed reality. Sightline cones show standing and seated views, and rules flag clearance, pitch and lux problems before hanging.
How we built it
Meta IWSDK, three.js and React Three Fiber, with IWER hand emulation. Twenty typed actions, four Gemini roles on Vertex AI, and a deterministic seven-rule engine own every verdict.
Challenges we ran into
Models propose plausible numbers that break rules, so verdicts had to be deterministic. Our own evaluation also exposed a placard action that reported success without saving text.
Accomplishments that we're proud of
Typed actions passed 16 of 16 frozen curator tasks against 2 of 16 with generic screen control, and the live Cloud Run build passes a real-browser suite with zero console errors.
What we learned
Give agents a small typed action surface and let a deterministic engine judge the result. Testing that surface in a real browser exposed bugs that unit tests mi
What's next for Sightline Studio
This is a demo product. Thus it has a monolithic architecture, uses synthetic data, it is not yet consumer ready. If I get a pilot I can then rewrite it with a proper long term easy to maintain architecture so that it can hold with ease years of incremental functionalities.
Built With
- express.js
- fov-aware-ui
- gemini-2.5-flash
- gemini-2.5-pro
- google-cloud-run
- hand-tracking
- hands-first
- iiif
- immersive-web-sdk
- iwer
- meta-quest-3
- node.js
- passthrough
- react
- react-three-fiber
- seated-design
- spatial-anchors
- spatial-audio
- three.js
- typescript
- uikitml
- vertex-ai
- vite
- webxr