FireSight

The FireSight command dashboard: a live 3D map of the field, with the drone camera, AI annotations and every glasses wearer in one view.
Man-down detection: after a hard fall, the glasses ask "Are you OK?". With no tap in 20s, a MAN DOWN alert with the wearer's exact location.
On the G2 glasses: the nearest fire's distance and direction, whether it's getting closer, and a mini-map with you at the center.
Live frames from a DJI drone, tracked in real time. AI marks fire and hazards and places them on the map with real coordinates.

Inspiration

Picture a hiker on a ridge trail when the smoke rolls in. Or a family leaving home as a wildfire jumps the road. Their questions are simple: Where is the fire? How far away is it? Is it getting closer? Usually, nobody can tell them.

Then we realized the people with the most at stake already have aerial views, and still face the same problem: firefighters. Drones fly over wildfires every day, but that view rarely reaches the person standing in front of the flames. Burnovers, where crews are caught by an advancing fire front, have historically caused about one in five wildland firefighter deaths. In those moments, a firefighter needs the same three answers as the hiker, instantly and without putting down their tools.

What exists today:

Drone feeds go to the command post. Incident commanders watch live aerial and infrared video on screens, but crews on the line usually hear about it secondhand over the radio.
Tracking and maps live on phones and tablets. The Forest Service adopted TAK (Team Awareness Kit) for firefighter tracking. In an early field pilot, positions updated about every two minutes, and using it meant stopping to look down at a screen.
Heads-up displays exist for structure fires. Helmet-mounted thermal displays help firefighters see through indoor smoke, but they cost thousands per unit and don't carry drone or map intelligence.

How FireSight is different:

The drone's view, in the wearer's line of sight. AI turns aerial footage into one glanceable line, e.g. "Fire ~160 m NE · Closing in", plus a mini-map with you at the center. No screen to check. Relative to you. Distances and directions are calculated from your GPS position and update as you move, so the information is about your situation, not just the fire.
Two-way. Your position, motion and status go back to the dashboard every 5 seconds. SOS and man-down alerts arrive with exact coordinates.
Consumer hardware. It runs on off-the-shelf smart glasses and works with existing drone setups, so the same system can serve firefighters today and hikers and residents tomorrow.

That's FireSight: the view from above, delivered to the people on the ground, eyes up and hands free.

What it does

FireSight connects a drone to smart glasses:

Sees from above: a drone streams live video. Our pipeline tracks the camera's position and geo-references each frame.
Understands the scene: AI (OpenCV and Google Gemini) identifies fire, smoke and hazards in the frames and places them on a real map with GPS coordinates.
Delivers it hands-free: Even Realities G2 smart glasses show short, glanceable updates:
- the nearest hazard's distance and direction (e.g. "Fire ~160 m NE")
- whether you're closing in or moving away
- a north-up mini-map with you at the center
Keeps people connected: the glasses report the wearer's GPS, motion and battery every 5 seconds to a live dashboard, so a coordinator can see everyone on the map.
Safety built in:
- Hold a temple → SOS, with your exact location.
- If the glasses detect a hard fall and you don't tap "I'm OK" within 20 seconds, they send a man-down alert with GPS.
- A tap acknowledges updates, so a coordinator knows they were seen.

How we built it

Drone and vision: a DJI drone streams over RTMP into Beagle Eyes, our visual-odometry pipeline (DPVO), which tracks the camera. A GPS transform module geo-references frames, and person detection runs alongside. Frames and telemetry are pushed to the backend.
Backend: Python/FastAPI on Render (Docker).
- Each frame is analyzed with OpenCV and Gemini VLM provided by Google AI studio.
- Operator-driven Gemini annotations mark fire, smoke and hazards, and are projected from image pixels onto latitude/longitude.
- It computes each wearer's distance and bearing to hazards and renders a 288×144 greyscale map sized for the glasses.
Dashboard: a map-first web app built with MapLibre: live drone view, AI annotations, every wearer as a live marker, and red SOS / MAN DOWN banners.
Glasses app: TypeScript + Vite with the Even Hub SDK. It runs in the Even phone app and drives the G2 display. It reads phone GPS, streams the glasses' accelerometer at 10 Hz, records which temple was pressed, and runs fall detection on the device.

Challenges we ran into

A tiny display: 576×288 pixels, 16 shades of green, about 9 short lines. Every word and pixel had to earn its place, so we built a north-up map renderer whose scale bar zooms as hazards get closer.
Undocumented hardware behaviour:
- Taps arrive through a different event type than the docs show.
- The startup page can only be created once.
- The IMU stays silent unless it's enabled after the display is created.
- We found each of these by testing on real hardware.
Calibrating fall detection from real data: we recorded the glasses' accelerometer while standing still, nodding, walking and taking them off. Removing the glasses spiked to 2.5 g, so we set the fall threshold at 3 g and required stillness afterwards, to avoid false alarms.
Single frames don't know where they are: turning a pixel into a real-world position required camera pose tracking plus GPS geo-referencing.
Hackathon realities:
- The AI free tier allowed about 20 requests a day, so we added per-image caching and an OpenCV fallback.
- We merged four people's branches into one deployed system.

Accomplishments that we're proud of

A complete loop running on real hardware: drone → AI → map → glasses → back to the dashboard.
Live distance and direction on the glasses that update as you walk, turning critical as you approach a hazard.
SOS and man-down alerts with GPS that reach the dashboard in seconds, tested with real drops of the glasses.
Fall detection tuned on real sensor recordings, not guesses.
Everything is deployed and running on Render, so the demo needs nothing but the glasses, a phone and a browser.

What we learned

Glanceable beats detailed. A distance, a direction and a trend ("closing in") are more useful at a glance than any amount of footage.
Test on hardware early. Several SDK behaviours only showed up on the real glasses.
State limits plainly in safety tools. We mark approximate positions, avoid claiming precision we don't have, and treat this as a prototype, not certified safety equipment.
How to project image detections into real-world coordinates, and how to design around a 16-shade, 576-pixel display.

What's next for FireSight

A consumer mode for hikers and residents: alerts that work even without a dedicated drone, using public fire-detection data alongside community drone feeds.
*Pilot with fire agencies: * Test FireSight with crews from agencies like the U.S. Forest Service and CAL FIRE, to learn what information actually helps under pressure, and refine the glasses display around it.
Better direction cues: "fire ahead-right" using the wearer's direction of travel, and calibrated drone positioning for tighter hazard locations.
Group awareness: see your group or crew on the glasses' map, plus a "check on someone" request.
Voice reports: hold to speak, transcribe, and feed what people on the ground see back into the AI's picture.
Resilience: offline caching and low-connectivity modes for areas where cell coverage drops.