4arms

Eyes on the beam, safety on screen: 4arms powers Rayban Meta Wayfarer (Gen 2) glasses with Amber AI for instant hazard alerts, auto-procurement, and site logistics

Comment

About 4arms — Physical AI Construction Platform

Inspiration

Infrastructure construction sites are high-stakes environments where split-second decisions dictate worker safety and million-dollar project timelines. Yet, despite rapid technological advancement, structural workers on beams and in hazardous zones operate with zero real-time AI assistance. All critical safety protocols, building models, and material manifests live inside documents workers cannot open when their hands are busy on site.

We built 4arms to change that. Inspired by the Physical AI Track challenge, we set out to bridge edge computer vision with project-wide cognitive intelligence. By mounting AI perception onto Meta Smart Glasses backed by NVIDIA edge compute, 4arms gives workers a "fourth arm" of intelligence—seeing what they see, forecasting logistics, and speaking up the moment safety or supply chain bottlenecks arise.

What it does

4arms is an end-to-end Physical AI platform operating across four critical site capabilities:

  1. Real-Time Safety Hazard Detection: Uses Meta Smart Glasses visual telemetry to identify unequipped workers, fall hazards, scaffolding issues, and machinery proximity. Displays heads-up warnings on the lens, triggers voice alerts, and dynamically computes a live job-site safety score $S_{\text{site}} \in [0, 100]$: $$S_{\text{site}} = \max\left(0, 100 - \left(15 \cdot N_{\text{critical}} + 10 \cdot N_{\text{high}} + 5 \cdot N_{\text{medium}} + 2 \cdot N_{\text{low}}\right)\right)$$

  2. Autonomous Procurement & Vendor Selection: Monitors site inventory levels $Q_{\text{stock}}$. When stock drops below critical threshold $Q_{\text{reorder}}$, 4arms auto-generates purchase orders and ranks potential vendors using a weighted ratio of historical reliability to unit price factor: $$V_{\text{score}}(v) = \frac{\text{Reliability}(v)}{\text{Price Factor}(v)}$$

  3. Last-Mile Delivery Routing & Dock Staggering: Uses the Haversine distance model coupled with real-time speed vectors $v_{\text{traffic}}$ to project truck arrival times (ETAs): $$\text{ETA (minutes)} = \max\left(10, \left(\frac{R_{\text{earth}} \cdot c}{v_{\text{traffic}}}\right) \cdot 60\right)$$ It then dynamically assigns staggered dock time slots to eliminate gate congestion.

  4. Fabrication Lead-Time & ROJ Date Forecasting: Combines baseline fabrication rates with stochastic variance $\delta \sim \mathcal{N}(0, \sigma^2)$ and risk multipliers $\gamma_j$ to predict Receipt on Job (ROJ) dates: $$L_{\text{pred}} = \left(L_{\text{base}} + \delta\right) \cdot \prod_{j \in \text{Active Risks}} \gamma_j$$

How we built it

  • Backend & Orchestration: Developed in Python 3.13 using FastAPI, Async SQLAlchemy, and SQLite. Built a modular event router (amber_agent.py) capable of handling simultaneous sensor inputs across all four capabilities.
  • Real-Time Streaming: Implemented WebSockets for zero-latency bidirectional communication between edge glasses simulators and the operational dashboard.
  • Day Worker Frontend: Created a high-contrast, light-themed responsive dashboard using HTML5, Vanilla CSS tokens, and Leaflet GIS maps, specifically engineered for outdoor readability under direct sunlight.
  • Edge Simulator Engine: Created an asynchronous Python telemetry emulator mimicking real-time Meta Smart Glasses camera bursts, voice commands, and spatial location signals.

Challenges we ran into

  • Python 3.14 Binary Compatibility: During environment provisioning, compilation errors arose due to missing pre-built wheels for underlying C/Rust extensions (pydantic-core, uvicorn). We rapidly solved this by migrating to uv and Python 3.13, accelerating package resolution to under two seconds.
  • Daylight Glare & Outdoor Ergonomics: Dark-mode UIs look sleek in office settings but fail on high-visibility glare construction sites. Re-engineering our entire design system to high-contrast light theme variables ensured critical red/amber hazard indicators remain instantly readable in harsh sunlight.
  • Staggering Dock Logistics Without Gate Queuing: Balancing vendor delivery ETAs with physical unloading zone capacities required creating a dynamic staggering offset algorithm to ensure trucks do not idle on perimeter roads.

Accomplishments that we're proud of

  • End-to-End Multimodal Integration: Built an operational, full-stack platform covering safety, procurement, last-mile delivery, and lead-time forecasting in a unified event loop.
  • Ultra-Fast Edge-to-Dashboard Sync: Achieved near-instantaneous WebSocket state synchronization from glasses simulation to the site operations center.
  • Hackathon-Ready Software Engineering: Delivered clean, modular code backed by automated database seeding, complete Git version history, and enterprise-grade system architecture documentation.

What we learned

  • Ergonomics Drive Physical AI Adoption: Building AI for physical workers requires prioritizing hands-free speech output, heads-up lens notifications, and glare-resistant visual interfaces over complex menu trees.
  • Supply Chains are Interconnected Systems: Safety delays on site directly cascade into delivery dock shifts and lead-time re-evaluations. Architecting 4arms around unified event routing proved essential.

What's next for 4arms

  • On-Glass Computer Vision Deployment: Quantize computer vision models to ONNX runtime for local inference on NVIDIA Jetson / Meta Smart Glasses hardware.
  • Direct 3D BIM Model Spatial Mapping: Overlay real-time hazard alerts and delivery unloading zones onto 3D IFC/BIM site models.
  • Autonomous Vendor RFQ Negotiation: Enable automated multi-tier approval workflows allowing low-value material purchase orders to execute autonomously within preset budget guardrails.

Built With

Share this project:

Updates

Submission history