Inspiration

VR headsets like the Meta Quest Pro already track your eyes many times per second, but that data is only used for rendering. Nobody uses it to tell you how a session is going for you. The usual check is a questionnaire at the end, which is late, subjective and easy to skip.

We wanted a measured signal, compared against your own starting point rather than a population average. We also wanted it built so it never pretends to know more than it measured.

What it does

VRFatiq runs a complete session on the Meta Quest Pro:

Baseline. A calm stretch that learns how your eyes normally behave.
Task. Focused work, with render settings locked to match the baseline.
Signals. Deterministic, on-device metrics plus an experimental change-from-baseline indicator.
Self-report. Two 1–7 sliders and quick chips. Free text never leaves the headset.
Report. Local rules compare your words with the signals and check for discomfort keywords. If you explicitly consent, the backend asks Gemini to explain the result.
Recover. Finish with the report, or step into a calming, parametric forest scene.

Every screen shows where its data came from (LIVE, REPLAY or SYNTHETIC) and where the explanation came from (live, cached or fallback).

The indicator measures change from the start of the session. It is not a medical fatigue score.

How we built it

Client: Unity 6 (URP, Vulkan), OpenXR, XR Interaction Toolkit and the Meta XR Core SDK, all in C#. We read gaze directly through OVRPlugin and fetch the head pose for the eye sample's own timestamp, not the current frame. We measure the sample rate from real intervals instead of assuming 90 Hz.

Signal core: pure C# with no Unity dependency, so the same code runs on the headset, on recorded replays and in tests. The pipeline is:

$$\text{eye samples} \rightarrow \text{quality} \rightarrow \text{metrics} \rightarrow \text{baseline} \rightarrow \text{indicator}$$

Quality gates: maximum gap of 100 ms, at least 1 s of valid time, coverage $\ge 0.8$. Invalid rows act as barriers, and we never interpolate across them.
Smoothing: a Savitzky–Golay filter on uneven timestamps, solved with re-orthogonalized QR.
Angles: computed in a numerically stable way as $\theta = \operatorname{atan2}\left(\lVert \mathbf{a}\times\mathbf{b}\rVert,\ \mathbf{a}\cdot\mathbf{b}\right)$. Head rotation uses quaternions that are invariant to the $q / -q$ sign ambiguity.
Robust baseline: 20-second windows, with at least 6 usable ones required. Each metric becomes a robust z-score:

$$z = \operatorname{clip}!\left(\frac{x - \operatorname{median}}{\max\left(1.4826\cdot\mathrm{MAD},\ \text{floor}\right)},\ -4,\ 4\right)$$

Indicator: a weighted combination of deviations in latency (weight 3), RMS error (2), miss rate (2) and p95 gaze speed (1). It is smoothed with an EWMA:

$$s_t = 0.3\, r_t + 0.7\, s_{t-1}$$

The thresholds are low $< 0.35$, moderate $< 0.6$ and high $\ge 0.6$. A break is suggested after 3 consecutive windows above $0.6$, with hysteresis.

Backend: Python 3.12, FastAPI and Pydantic v2 in strict mode, packaged with Docker (non-root, locked dependencies via uv) and deployed on Railway.

Gemini can only choose from an approved vocabulary of codes and bounded scene parameters; the app writes the actual text.
Requests use bearer-token auth with constant-time comparison and byte-accurate body limits.
Error responses never echo the submitted values.
A worst-case cost budget is tracked in SQLite.
Answers are cached in an LRU cache keyed by SHA-256.

Contracts: 14 versioned JSON Schemas are the single source of truth for C#, Python and tests. GitHub Actions runs tests, Ruff, contract checks and a Docker build on every push.

Tooling: we used Claude Code and Cursor to move fast across about 16k lines of C# and 4k lines of Python, and Grok for research and brainstorming.

Challenges we ran into

Real eye data is messy. We had to handle duplicate and out-of-order timestamps, dropped frames, stalls, tracking loss and one eye at zero confidence. We designed for all of it explicitly instead of hoping it wouldn't happen.
Separating fatigue from rendering artifacts. Foveation, render scale, MSAA or a refresh-rate change can alter eye behavior on their own. We lock all of them between baseline and task; if any changes, the comparison is cancelled.
Testing eye tracking without living in the headset. We built a deterministic 190-second synthetic session that reproduces real headset artifacts, plus 8 anomaly fixtures, each with a golden quality report.
Making AI safe in a health-adjacent context. We constrained Gemini to approved outputs, validated every response, and fall back to a fully local report when needed.
Keeping three codebases in sync. The chip-to-scene rules live in Unity, the backend and the forest scene, and all three are checked against one shared scene_rules.json.

Accomplishments that we're proud of

450+ automated tests across Unity EditMode, PlayMode and backend pytest, plus Python validators that independently recompute medians, MAD and window validity.
Real tests on Meta Quest Pro passed:
- 60-second live eye-stream diagnostic;
- axis verification;
- gaze staying on a fixed target while the head turns;
- permission denial and revocation;
- recenter and pause handling;
- the full session end to end.
A "never make up a number" design.
- A missing metric makes the indicator report unavailable instead of quietly re-weighting the rest.
- Every boundary fails closed.
- Missing eye permission never silently becomes fake data.
Privacy by default. Free text and raw eye data never leave the device, and the cloud is opt-in per session.

What we learned

Eye tracking is a timing problem as much as a geometry problem. Timestamps, sample rates and head-pose alignment matter more than any single formula.
Robust statistics (median, MAD, clipping) beat clever models when the data is noisy and small.
Strict contracts between languages save hours of debugging.
AI is most useful when it is bounded: choosing from safe options is far easier to trust than free generation.
Synthetic tests let you move fast, but nothing replaces putting the headset on.

What's next for VRFatiq

Better models: learn the indicator's weights and thresholds from real session data instead of engineering defaults.
More precise math: per-person gaze-accuracy models and tighter filters. Quest Pro accuracy varies between people, with a median around $1^\circ$ and heavy tails.
Measured calibration: verify eye calibration from the data instead of relying on operator confirmation.
A larger user study: validate the indicator against self-reports across many users and sessions.
The full relaxation forest: ship the complete 68-model parametric scene with lighting, fog, layered ambient sound and a friendly squirrel.