Inspiration

Inspiration

Weather is one of the most universal metaphors for human emotion, we feel foggy, electric, calm after rain, or warm as golden hour. VieWs began as a flat 2D iOS/Swift concept exploring affective journaling, where words shifted colors on a screen. When we stepped into Meta Quest 3, we realized emotions shouldn't be trapped inside a 2D rectangle: you should be able to sit inside your feelings, part the air of your physical room with your bare hands, and watch your words take physical shape around you as living 3D weather, spatial sound, and light.

What It Does

VieWs XR is a controller-free, hands-first affective weather sanctuary built natively on the 100% official Meta Horizon OS + Unity 6 OpenXR stack:

  1. Spoken Spatial Onboarding (Wit.ai TTS & Passthrough): Launches in Mixed Reality Passthrough (OVRPassthroughLayer) with spoken spatial guidance rendered via Meta Wit.ai TTS (wit$Railey) that previews gentle fog and rain right inside your room.
  2. Voice-to-Atmosphere & Overlay Keyboard (AppDictationExperience + Meta AI Cloud): Speak how you feel using Meta Voice SDK / Platform Dictation (Oculus.Voice.Dictation.AppDictationExperience + Wit.ai fallback) or pinch the minimal floating prompt to type with the native Meta Quest System Overlay Keyboard (oculus.software.overlay_keyboard).
  3. Meta AI Building Blocks (LlmAgent + Llama Cloud API): Your feeling is interpreted by Meta AI Building Blocks (Meta.XR.BuildingBlocks.AIBlocks.LlmAgent) into a structured 3D atmospheric state (WeatherState) and a 5-swatch emotional color palette (ColorPalette)—driving precipitation, snow, volumetric fog, wind vectors, lightning frequency, color temperature, and custom procedural particle geometry (soft_orb, needle, ribbon_streak, crystalline_hex, origami_petal, ember_shard).
  4. Two-Hand Palm Portal (PalmPortalGesture & PalmReturnGesture): Once your sky is ready, face your palms together and gently pull them apart in mid-air to tear open a spatial portal from your physical room into a 360° VR terrain and mirror-pond sanctuary. When you're ready to return to your room, bring your open palms back together to seal the portal back to Mixed Reality.
  5. Physical Hand-to-Weather Interaction (HandWeatherForceField): Reach out with tracked hands (OVRHand + OVRSkeleton): sweeping an open palm parts and repels nearby raindrops, snowflakes, and embers, while pinching creates a localized vortex swirl and holding your palm upward reveals a floating poetic inspection card.
  6. 3D Glass Memory Orbs (MemoryOrbConstellation): Every generated atmosphere is saved as a custom-shader refractive glass orb (VieWsXR/MemoryOrbGlass) floating in a seated arc. Aim and short-pinch an orb via RayInteractable to restore that memory, or hold for 0.75s to dissolve and delete it.

How We Built It (Complete Re-Engineering onto the Full Meta XR Stack)

We performed a complete ground-up re-engineering into C#, Unity 6 (6000.0.85f2 URP), and the full Meta XR SDK v209 + Meta Voice SDK v85 stack:

  • Meta Interaction SDK (com.meta.xr.sdk.interaction.ovr): Integrated OVRComprehensiveInteractionRig with PointableCanvas, PointableCanvasModule, ClippedPlaneSurface, ColliderSurface, and RayInteractable for 100% controller-free point-and-pinch navigation across all floating stages and 3D memory orbs, while keeping custom OVRHand/OVRSkeleton bone tracking active and disabling duplicate hand meshes.
  • Meta AI Building Blocks & Zero-Secret Credential Architecture: Implemented Meta.XR.BuildingBlocks.AIBlocks.LlmAgent with a custom MetaCloudLlmProviderAsset and CredentialStorage (Assets/Settings/AIProviders/MetaCredentialStorage.asset). Zero API keys are baked into the repository or APK; the app uses a secure guest session flow with live runtime BYOK override in the in-headset Meta AI drawer, plus an honest deterministic OfflineSanctuary mode when offline.
  • Meta Voice SDK & Wit.ai (com.meta.xr.sdk.voice): Wired AppDictationExperience with Quest Platform Dictation (UsePlatformIntegrations = true) and Wit.ai speech-to-text/text-to-speech (Assets/witai.asset) for hands-free voice prompts and spatial voice guidance.
  • Meta Passthrough, MRUK & Spatial Audio (com.meta.xr.sdk.core, com.meta.xr.mrutilitykit, com.meta.xr.sdk.audio): Built seamless VR <-> MR skylight blending over OVRPassthroughLayer and a procedural DSP AudioEngineService that synthesizes real-time rain, wind, thunder, and harmonic chimes with automatic narration ducking.

Challenges We Ran Into

  • Coordinating Simultaneous Hand Rays &Two-Hand Gestures: Keeping Meta's OVRComprehensiveInteractionRig ray-pinch UI clean without double-rendering OVRHand meshes or accidentally firing portal return gestures while a user is pinching a UI button or memory orb required careful pointer-ID isolation, gesture gating, and focus-loss cancellation (OVRManager.InputFocusLost).
  • Natural Asymmetric Two-Hand Portal Physics: Real human hands rarely move at identical speeds. We engineered PalmPortalGesture and PalmReturnGesture to track palm normal facing, independent left/right seam edges, and brief tracking-loss tolerance so opening and closing the world feels tactile and forgiving.

What We Learned & What's Next

Building for Horizon OS showed us how powerful minimalist, borderless spatial typography is when paired with physical hand gestures and spoken audio cues. Next, we plan to expand MRUK room-mesh collisions so rain and snow physically collect on real-world desks and windowsills, and add shared multi-user co-located weather sanctuaries.

Built With

  • c-sharp
  • hand-tracking
  • hlsl
  • horizon-os
  • llama
  • meta-ai
  • meta-interaction-sdk
  • meta-quest-3
  • meta-voice-sdk
  • meta-xr-sdk
  • mixed-reality
  • mruk
  • openxr
  • passthrough
  • shaders
  • spatial-audio
  • unity
  • unity-6
  • urp
  • wit-ai
Share this project:

Updates

Submission history