RayCon

Inspiration

We didn't start with the hackathon brief. We started with a harder problem: physical AI has no internet-of-text to learn from — humanoid robots and automation need embodied, egocentric data that simply doesn't exist yet, and we'd already spent months thinking about how to capture it. So when we read Track 1's framing — "everything a worker needs to know lives in a document they can't open with their hands full on a beam" — it landed differently than a typical hackathon prompt. It wasn't just a construction problem. It was the exact embodiment gap we'd already been building toward, pointed at a place where getting it wrong costs lives. OSHA's Fatal Four — falls, struck-by, caught-in/between, electrocution — cause over 60% of construction deaths every year, and almost none of them are a knowledge problem. Workers usually know the general danger of a live wire or an unshored trench. What kills them is the specific moment: a hazard that's been sitting in their peripheral vision for ten seconds while their attention is somewhere else. Existing safety cameras can log that a hazard existed. None of them can tell you whether the person actually saw it. That gap became the whole project.

What it does

RayCon is a safety copilot built for Meta smart glasses. It sees exactly what the worker sees, and it runs a continuous loop: see the site → understand the moment → speak up only when it matters. It detects the hazards named directly in the brief — unprotected edges without a harness clipped in, exposed wiring near a live panel (escalated further if conditions are wet), unguarded rotating machinery, unshored trenches within arm's reach, a worker crossing into a crane's swing radius, scaffolding missing base plates — alongside standard site perception: workers, vehicles, cranes, tools, and PPE (treated as a supporting signal, never the headline). The core differentiator is attention-aware escalation. Every tracked hazard is checked against the worker's head orientation. If they've clearly looked straight at it, the system backs off — no nagging. If it's sat unnoticed for several seconds, the alert escalates, calmly at first and more urgently if it's still being ignored. It's the difference between a system that treats every hazard the same and one that knows whether it's already been handled.

How we built it

We split the build into tiers on purpose, so we'd always have something real to show even before we had the actual glasses hardware. Tier 1 (working now): YOLO26 for detection, BoT-SORT with re-identification for persistent multi-object tracking, YOLO26-Pose for fall detection and head-yaw estimation (chosen over MediaPipe specifically because its cost doesn't scale with the number of people in frame — a real constraint on a busy site), and a monocular depth model for distance-and-direction voice alerts. All of it runs locally on Apple Silicon via MLX, with PyTorch MPS as a fallback. Tier 2 (in progress): an async reasoning layer using Gemma 4 via local Ollama, fed by a NetworkX scene graph rather than raw frames, so it can reason about task, progress, and predicted hazards without competing with Tier 1's frame-rate loop. Tier 3 (the plan): once we're through Stage 1 and have actual glasses and NVIDIA edge compute, the same pipeline moves onto a phone-bridge-to-Jetson architecture, working toward a live site digital twin using RGB-only Gaussian Splatting SLAM — since the glasses have no depth sensor — and chunk-based memory management to keep that mapping tractable at real construction-site scale.

Challenges we ran into

The hardest problems weren't the obvious ones. There's no good open dataset for something like "exposed live wiring" — we had to design an honest proxy instead of pretending it was solved. Head yaw is a proxy for attention, not real gaze tracking, and we had to be deliberate about where that approximation could fail quietly. Depth estimation needed a per-camera calibration constant that doesn't generalize on its own. And partway through, we discovered Meta's glasses run no third-party code on-device at all — every frame has to bridge through a phone before it ever reaches real compute, which reshaped our whole Tier 3 architecture. We also had to resist the instinct to over-promise: it was tempting to describe every tier as "done," and we made a rule early on to flag, in plain language, what's real versus what's a scoped-down stand-in for the demo. The Major issue was in running the VLM to process the information fast, it takes a many seconds instead of being realtime which is required.

Accomplishments that we're proud of

Most of all: we didn't wait for the hardware. The core Tier 1 pipeline — detection, tracking, gaze-aware escalation, voice alerts — was built and tested end-to-end on a laptop camera before Stage 1 even closed. Handing us the glasses doesn't start our build; it moves an already-working system onto the worker's face. We're also proud that privacy was designed in from day one — face blurring in line with India's DPDP Act, session-scoped tracking with nothing persisting past a shift — instead of treated as an afterthought once judges asked about it. The diffrentation plan of implementing community based common world.

What we learned

The technical lesson was that model choice is never neutral: the difference between YOLOv8 and YOLO26, or MediaPipe and YOLO-Pose, isn't cosmetic — it's the difference between a system that holds up in a crowded, multi-worker scene and one that doesn't. The bigger lesson was about trust. A safety system that cries wolf gets muted or switched off by the people wearing it, and that failure mode matters more than any accuracy metric — which is exactly why the attention-aware logic, not just raw detection, became the heart of the project.

What's next for RayCon

Finish Tier 2 — task recognition, blueprint and procedure retrieval, predictive hazards grounded against real site documents. Get RayCon onto actual Meta glasses and NVIDIA edge compute the moment Stage 1 shortlisting comes through, and validate the phone-bridge pipeline against real Bluetooth/Wi-Fi latency instead of our simulated version. Then build toward Tier 3: a live, site-wide digital twin with multi-worker shared memory, so a crane operator's system can alert every nearby worker's glasses directly, not just their own — turning individual protection into a shared one.