Afferent-Omni

• 

  Afferent-Omni: Causal Vision-to-Touch Prior for Physical AI

💡 Inspiration

In modern dexterous robotics, vision has solved perception, but touch remains the missing link. While robotic arms can accurately locate objects visually, handling delicate materials, deformable elastomers, or sliding surfaces requires immediate, sub-millisecond tactile feedback. Traditional systems often couple heavy AI reasoning directly with low-level hardware control, introducing dangerous latencies.

Afferent-Omni was inspired by biological sensory-motor loops—where tactile reflexes happen instantly at the spinal cord level, while high-level cognitive reasoning occurs asynchronously in the brain. Our goal was to bridge this gap for the Nebius x NVIDIA Global AI Hackathon (Physical AI Track) by creating an enterprise-ready, causal vision-to-touch prior system.

🛠️ How We Built It

Afferent-Omni is structured around a robust, containerized architecture that decouples deterministic edge safety from cloud intelligence:

• Dual-Tier Architecture & Edge Control: Built with FastAPI powering 9 routers and 17 active endpoints, handling deterministic 50 Hz edge safety control and a neuromorphic reflex arc operating at sub-10 ms latency.
• Cloud AI Reasoning (NVIDIA & Nebius): Integrated NVIDIA Nemotron 3 Super 120B via Nebius Token Factory to process live telemetry JSON, providing real-time safety assessments and material analysis.
• 3D Frontend & Visualization: Developed using React Three Fiber and Three.js to render real-time 32×32 elastomer stress grids, glow spheres, and force vector fields.
• Tribology & Acoustics: Features an acoustic-slip spectrogram capturing harmonic micro-vibrations at 314 Hz.
• Compliance & Auditing: Automated ISO 10218-1 and ISO/TS 15066 compliance checks that export official audit PDF reports alongside 50 Hz Session Replay capabilities (JSON/CSV).

📐 Mathematical Framework & Causal Prior

At the core of our vision-to-touch prior, elastomer deformation elastomer stress tensor \( \sigma_{ij} \) is modeled over a contact patch governed by Young's Modulus \( E \) and friction coefficients \( \mu \):

$$ \sigma_{ij} = C_{ijkl} \varepsilon_{kl} - \int_0^t \Gamma(t - \tau) \frac{\partial \varepsilon_{kl}}{\partial \tau} d\tau $$

The Sim2Real Safety Shield validates grasp authorization based on uncertainty bounds \( \mathcal{U} \):

$$ \mathcal{P}(\text{Slip}) = 1 - \Phi\left( \frac{F_{\text{threshold}} - F_{\text{contact}}}{\sigma_{\text{uncertainty}}} \right) $$

⚙️ Challenges We Faced

1. Edge-Cloud Latency Decoupling: Balancing heavy LLM token generation via cloud APIs with a strict 50 Hz deterministic safety loop required careful asynchronous pipeline design using WebSockets.
2. Real-time 3D Elastomer Stress Rendering: Optimizing the Three.js 32×32 grid to update smoothly alongside high-frequency telemetry streams without stuttering the UI.
3. Strict Compliance Formatting: Automating ISO standard checks into structured, downloadable audit reports while maintaining clean backend separation.

🚀 What We Learned

We mastered the integration of cutting-edge infrastructure—leveraging Nebius AI Cloud and NVIDIA Nemotron for low-latency intelligent reasoning over structured JSON telemetry. We also deepened our understanding of tribological tactile sensing, Sim2Real safety boundaries, and the importance of enterprise-grade containerization (Docker) for robotics software deployment.

Built With

• docker
• fastapi
• iso
• nebius-token-factory
• nvidia-nemotron
• python
• react-three-fiber
• websockets