Cognitive Gravity: Agentic Volumetric Synthesizer

A hands-first spatial workspace where onboard AI turns raw voice and text streams into kinetic knowledge spheres you collide, stretch, and synthesize using natural hand gestures.

Comment

Cognitive Gravity

Inspiration

Flat 2D interfaces compress complex multi-dimensional knowledge graphs into rigid, linear layouts. We wanted to break out of flat screens and make synthesis tangible—allowing creators and systems engineers to physically touch, manipulate, and collide knowledge nodes in 3D space.

Cognitive Gravity was born from a simple question: What if high-dimensional conceptual relationships possessed physical mass and spatial attraction? By pairing WebXR spatial hand-tracking with Meta Llama 3.3 70B, we created an environment where pinching, pulling, and merging knowledge nodes triggers real-time volumetric AI synthesis.

What it does

Cognitive Gravity is a WebXR spatial computing engine and volumetric knowledge synthesizer that renders high-dimensional concepts as interactive 3D vector nodes suspended in a virtual workspace. Powered by bare-hand tracking and real-time AI vector reasoning, it allows users to physically touch, manipulate, and collide ideas in mid-air.

  • Spatial Gesture Tracking: Using bare-hand tracking in Meta Quest browsers, users select, translate, and interact with knowledge clusters without physical controllers.
  • Pinch-to-Synthesize Engine: Bringing two knowledge nodes within a 4 cm threshold triggers a spatial synthesis event.
  • Llama-Powered Volumetric Reasoning: The backend processes node payloads via Meta Llama 3.3 70B, deducing latent connections, resolving contradictions, and dynamically instantiating child nodes in 3D coordinate space.
  • Visual Spatial Debugging: Real-time visual feedback spheres and coordinate anchors provide immediate tactile visual confirmation during gesture detection.

How we built it

cognitive-gravity/
├── cognitive-gravity-webxr/       # Three.js / TypeScript WebXR Spatial UI
└── cognitive-gravity-backend/     # FastAPI / Python 3.11 / Cloud Run Core
  • Frontend Engine: Built using Three.js, TypeScript, and the WebXR Device API. Hand tracking continuously computes distance vectors between the index fingertip and thumb tip. A synthesis event is triggered when the spatial distance $d$ falls within the pinch threshold:

$$d(\mathbf{p}_1, \mathbf{p}_2) = \sqrt{(x_2 - x_1)^2 + (y_2 - y_1)^2 + (z_2 - z_1)^2} \le 0.04\text{ m}$$

  • Backend Infrastructure: Developed with Python 3.11 and FastAPI, containerized using Docker, and deployed serverless to Google Cloud Run.
  • AI Orchestration: Leverages Meta Llama-3.3-70B-Versatile through Groq's high-throughput infrastructure to deliver sub-second structured JSON responses containing spatial coordinates and node text payloads.

Challenges we ran into

  • Hand Tracking Noise vs. Intentional Gestures: WebXR hand tracking introduces micro-jitters. Fine-tuning the gesture engine to distinguish between accidental hand movement and intentional synthesis pinches required applying exponential moving average (EMA) smoothing to continuous spatial coordinates.

  • Sub-Second Spatial Latency: Spatial VR demands near-instantaneous feedback. Any latency spike breaks immersion. We minimized round-trip time by optimizing FastAPI request parsing and leveraging Groq for ultra-low latency inference.

  • Monorepo Version Parity: Integrating the WebXR frontend and FastAPI backend into a clean monorepo required structuring git remotes carefully to preserve history without file collision.

Accomplishments that we're proud of

  • Butter-Smooth WebXR Performance: Maintaining a locked 90 FPS rendering target while calculating real-time spatial node bounds and gesture math.
  • Zero-Cold-Start Cloud Infrastructure: Containerizing and deploying a lightweight Docker image (python:3.11-slim) to Google Cloud Run with instant warm scaling.
  • Strict Meta Llama Architecture: Successfully engineering prompt constraints that force Llama 3.3 70B to reliably emit valid spatial node topologies and 3D coordinate placements.

What we learned

  • Designing for Spatiality: 3D user interfaces require completely unlearning 2D UI paradigms. Spatial depth, physical proximity thresholds, and clear visual cues dictate user confidence in virtual environments.
  • Structured Output Validation: Forcing LLMs to produce deterministic 3D coordinate vectors requires tight JSON schemas and unambiguous system prompts.

What's next for Cognitive Gravity: Agentic Volumetric Synthesizer

  • Persistent Spatial Vector Store: Integrating Supabase (pgvector) to store high-dimensional graph embeddings permanently across sessions.
  • Agentic Sub-Graph Traversal: Evolving the synthesis engine from passive response generation into autonomous AI agents that proactively branch, test, and reorganize spatial knowledge clusters.
  • Multi-User Collaborative VR: Adding WebSocket spatial anchor synchronization so remote teams can collaborate in shared volumetric knowledge environments.

Built With

Share this project:

Updates

Submission history