Project Story

## Inspiration

Data is everywhere, but answers are stuck behind experts.

On a factory floor, the people who could act on data — supervisors, operators, maintenance leads — usually can't query it themselves. They wait for an analyst or a weekly report, and by then the moment has passed. Dashboards help, but they only answer the questions someone thought to build in advance, and they still need training to read.

The problem goes deeper than the workplace. In S&P Global's worldwide financial-literacy survey, roughly 40% of U.S. adults got the basic compound-interest question wrong — the same concept behind every savings plan, loan, and mortgage. If people can't grasp data on a flat screen with time and training, we need a fundamentally different way to let them interact with it.

Three shifts made that possible right now:

  • Affordable standalone headsets with built-in hand tracking (Meta Quest) — no PC tether, no controllers
  • AI models that can write reliable analysis code from a plain-English question
  • Speech recognition that's cheap and accurate, making voice a practical input when your hands are busy

PRISM combines all three: ask your data a question out loud, get a plain-English answer, and step inside the chart.

## What it does

PRISM is a voice-first mixed reality data assistant for Meta Quest.

  1. Ask — pinch and hold with either hand, speak naturally: "Which machine uses the most energy?"
  2. Understand — PRISM answers in plain sentences and reads them aloud
  3. See — a clear chart appears in the chat panel
  4. Explore — open it as a 3D graph in the room: touch a bar for details, grab it, zoom in

It currently ships with two demo experiences:

  • Factory Insights — a 1,000-job manufacturing dataset covering machines, operations, materials, energy use, and processing time. Ask for bar, pie, line, or scatter charts and get grounded answers.
  • Savings Coach — a compound-interest "what if" explorer. Ask "What if I save $200 a month for 20 years?" and watch 3D bars grow year by year — blue for contributions, amber for interest earned.

Trust is built in. PRISM never asks the AI to invent numbers. The AI writes analysis code; that code runs on the real dataset; the chart and the 3D graph are drawn from those exact results. Financial calculations don't use AI at all — they're computed directly.

## How we built it

Frontend — Unity 6 on Meta Quest

  • Hand tracking via the XR Interaction Toolkit and XR Hands packages — pinch-to-talk for voice input, fingertip touch for 3D graph interaction
  • Custom 3D graph rendering for bar, line, pie, and scatter charts with touch-to-inspect, pinch-to-move, and two-hand-pinch-to-zoom
  • A chat panel with speech bubbles, typing indicators, and inline chart cards
  • Voice routing that classifies each question and sends it to the right handler

Backend — Python Flask server

  • Google Gemini classifies questions and generates pandas analysis code
  • The generated code executes against the real CSV dataset — the AI proposes, the data decides
  • OpenAI Whisper handles speech-to-text, with automatic Gemini fallback
  • Charts are rendered server-side and sent as images; 3D graph data is normalized and sent as JSON
  • The Quest and PC communicate over the local network

The key architectural decision: separating who writes the analysis (AI) from who computes the numbers (code running on real data). This means every chart is grounded in fact, and the 2D chart and 3D graph always show the same numbers.

## Challenges we faced

Pinch conflicts. Pinch-to-talk and pinch-to-grab-the-3D-graph use the same gesture. We solved this with a HandZone system — when your hand is near or on a 3D graph, pinch gestures are routed to manipulation (move/zoom) and voice recording is suppressed. Step away, and pinch-to-talk works again. It sounds simple, but getting the spatial boundaries and state transitions right took significant iteration.

Trusting AI-generated analysis. Early versions let the AI answer data questions directly, and it would confidently hallucinate statistics. The fix was architectural: the AI only writes pandas code, and that code runs on the actual dataset. If the code errors out, PRISM says so — it never falls back to making numbers up. This "AI writes code, code computes numbers" pattern turned out to be the single most important design decision.

Networking reliability. The Quest and the PC must be on the same Wi-Fi network, and firewalls, VPNs, and changing IP addresses all cause silent failures. We added a /health endpoint for quick diagnostics and clear error messages that tell the user exactly what to check. Zero-setup networking (mDNS/auto-discovery) is on the roadmap.

Making 3D charts actually useful, not just cool. It's easy to make a 3D bar chart that looks impressive but is harder to read than a 2D one. We focused on interaction: touch a bar and a card shows its value, rank, percentage share, and a short AI-generated insight. The 3D view isn't a replacement for the 2D chart — it's an exploration layer on top of it.

## What we learned

  • Voice changes everything about UX. When the primary input is speech, you design for conversation, not for menus. The routing system that decides "is this a data question, a savings what-if, or a general question?" became central to the experience.
  • Spatial data has real advantages for non-experts. In informal testing, people who struggled to read a 2D pie chart immediately understood proportions when they could walk around a 3D version and touch the slices.
  • Hand tracking is mature enough for production input. Pinch-to-talk felt natural to every tester within seconds — no tutorial needed.
  • Grounding AI output in computation is non-negotiable. Users trust PRISM because it can show its work. That trust disappears the moment a number is wrong.

## What's next

  • Bring your own data — connect any CSV, Excel, or database
  • Zero-setup networking — the headset finds the server automatically
  • Shared sessions — multiple people exploring the same 3D graph together
  • More devices — Apple Vision Pro, Android XR, and a desktop companion

Built With

Share this project:

Updates

Submission history