Pscouter

Know those around you and gauge their power level. Know your place and know their place.

Comment

Saiyan Scouter XR: AI-Powered Pose Estimation for Immersive VR Experiences

Inspiration

Inspired by Dragon Ball's iconic power-scanning device, we created an immersive XR experience that combines cutting-edge AI pose estimation with intuitive VR interaction. Just like Goku's scouter detects power levels, our system analyzes human movement in real-time to provide meaningful feedback in virtual reality.

** What it Does**

Our project delivers a complete XR Pose Estimation Pipeline for PICO VR headsets that enables:

  • ** Real-time Pose Detection**: Uses MediaPipe models with Unity Barracuda for 17-point skeletal tracking
  • Intelligent Power Analysis: Converts pose quality metrics into "power levels" with Dragon Ball-inspired scaling
  • Immersive VR Interface: Floating 3D UI that follows users and provides real-time feedback
  • Modular Architecture: Extensible pipeline supporting multiple AI backends (Barracuda/ONNX, SecureMR/DLC)

Key Features

AI-Powered Pose Estimation

  • MediaPipe Integration: State-of-the-art 17-point pose detection
  • Barracuda Backend: Cross-platform neural network inference in Unity
  • SecureMR Support: PICO-optimized DLC models for maximum performance
  • Real-time Processing: Sub-100ms inference on PICO hardware

Dragon Ball-Inspired Interface

  • Dynamic Power Levels: Pose strength translates to combat power (5-10,000+)
  • Classic Scouter Design: Retro green interface with authentic sound cues
  • Auto-Targeting: Automatically detects and tracks pose presence
  • "OVER 9000" Mode: Special handling for exceptional poses

XR-First Architecture

  • MRC Camera Integration: Mixed Reality Capture for seamless AR/VR blending
  • 3D Spatial UI: Interface floats in 3D space, always facing the user
  • PICO Optimized: Native support for PICO XR SDK features
  • Modular Design: Easy extension for fitness, gaming, or medical applications

Technical Implementation

Core Technologies

  • Unity 2022+ with Universal Render Pipeline
  • Barracuda 3.0 for neural network inference
  • MediaPipe Pose Estimation models
  • PICO XR SDK for native VR support
  • C# Architecture with clean separation of concerns

Pipeline Architecture

📷 MRC Camera →  Pose Detection →  Power Calculation → VR Interface

Key Components

  • IPoseEstimator Interface: Unified API for different AI backends
  • SimplePoseEstimator: Barracuda/ONNX implementation
  • SecureMRPoseEstimator: PICO-optimized DLC implementation
  • SaiyanScouterXR: VR interface with Dragon Ball theming
  • Setup Helpers: Automated project configuration tools

** Challenges**

Technical

  • Package Resolution: Complex Unity package management for XR development
  • Performance Optimization: Real-time inference on mobile VR hardware
  • Cross-Platform Compatibility: Supporting multiple AI backends
  • MRC Integration: Seamless mixed reality camera access

UX

  • Intuitive Feedback: Translating technical pose data into engaging user experience
  • Spatial UI Design: 3D interface that doesn't break immersion
  • Real-time Responsiveness: Instant feedback for pose changes

Future Applications

Fitness & Training

  • Real-time form correction for workouts
  • Progress tracking with gamified power levels
  • Virtual personal trainers with pose analysis

Gaming & Entertainment

  • Motion-controlled combat systems
  • Dance games with pose accuracy scoring
  • Social VR experiences with pose-based interactions

Technical Innovation

  • First Barracuda + PICO XR Integration: Pioneering AI inference in VR
  • Performance Optimizations: Real-time inference on constrained hardware

User Experience Innovation

  • Gamified Pose Feedback: Making exercise data engaging and fun
  • Immersive UI Design: VR interfaces that enhance rather than distract
  • Accessible AI: Bringing computer vision to mainstream VR users

** Why This Matters**

This project bridges the gap between cutting-edge AI research and accessible VR experiences. By making pose estimation both powerful and easy to use, we open up new possibilities for VR interaction, fitness tracking, and immersive gaming.

Built With

  • aftereffects
  • cursor
  • figma
  • unity
Share this project:

Updates