Pyrosense (UC Berkeley )

Firefighter brain + AI: PyroSense fuses EEG (P300) with drone vision to surface hidden victims and hazards in smoke, prioritizing critical frames and guiding faster, safer rescue decisions.

Comment

PyroSense

A cognitive + computer vision system that converts firefighter intuition into real-time autonomous insight

The Problem

Modern firefighting operations increasingly rely on drones and computer vision systems to assess hazardous environments before human entry. These systems are valuable, but they break down precisely in the environments where they matter most.

Fire scenes introduce extreme perceptual challenges:

  • dense smoke that obscures edges, contours, and depth
  • rapidly changing illumination from flames
  • thermal reflections and heat bloom that distort true signal
  • partial occlusion of victims and hazards
  • chaotic, unstructured environments unlike training datasets

Computer vision models, even state-of-the-art architectures like YOLO, are fundamentally trained on relatively clean, well-lit datasets. When deployed in fire conditions, performance degrades due to the real-world conditions.

Prior work in adverse vision conditions shows that:

  • smoke and haze reduce contrast and degrade feature extraction, directly impacting detection pipelines
  • occlusion significantly lowers recall in object detection tasks
  • models trained on standard datasets struggle to generalize to thermal distortion and dynamic lighting

In practice, this means:

  • victims can be partially visible but not detected
  • hazards like structural collapse zones or hidden fire spread are missed
  • false positives increase, overwhelming operators

At the same time, experienced firefighters and operators often report:

“I saw something, but I couldn’t react fast enough.”

This is where subconscious response identification is useful

The P300 response is a well-documented EEG signal that occurs approximately 300 ms after a person detects something salient or meaningful, consciously or subconsciously. It is widely used in:

  • target detection experiments
  • lie detection paradigms
  • rapid serial visual presentation (RSVP) systems

Critically:

  • the P300 can occur before conscious decision-making
  • it reflects subconscious recognition of important stimuli

This creates a fundamental mismatch:

  • Machines scale but miss edge cases
  • Humans detect edge cases but cannot act fast enough

Our Solution

PyroSense is a cognitive-autonomy layer that fuses human subconscious perception with computer vision to create a real-time salience detection system for firefighting.

Instead of treating the human as a controller, we treat the human brain as a high-bandwidth sensor.

System behavior:

  • A firefighter observes a live drone feed
  • Their brain reacts to subtle anomalies or important patterns
  • EEG captures that response in real time
  • The system aligns that signal with the exact video frame
  • Computer vision analyzes that frame more deeply
  • The system surfaces:
    • victims
    • hazards
    • structural risks
  • A prioritized rescue plan is generated

This allows the system to:

  • detect what CV misses
  • act faster than human reaction time
  • reduce cognitive overload

Core Insight

The brain detects meaning before the human reacts
We convert that detection into machine-readable signal

PyroSense is not mind reading.
It is salience detection, grounded in measurable neural responses.

System Architecture

PyroSense consists of two tightly integrated subsystems:

1. EEG-Based Salience Detection System

We use the Muse 2 EEG headset to capture neural signals in real time.

Signal Acquisition Pipeline

  • Muse streams EEG via muselsl (Lab Streaming Layer)
  • muse_lsl_bridge.py:
    • reads LSL stream
    • converts timestamps into browser-aligned epoch milliseconds
    • computes rolling bandpower
    • forwards:
    • raw EEG chunks
    • bandpower summaries
  • Node server:
    • receives EEG packets
    • broadcasts via WebSockets
  • Frontend:
    • maintains a rolling EEG buffer synchronized with stimulus timestamps

Stimulus Collection System (/collection)

We built a controlled RSVP-style training and validation environment.

Image Categories:

  • Humans in fire (critical targets)
  • Items in fire (secondary hazards)
  • Normal / abnormal items (low salience baseline)

Trial Structure:

Each trial consists of:

  1. Fixation cross (500 ms)
  2. Image display (300–500 ms)
  3. Blank interval (700–1000 ms)

At exact image onset, the system records a timestamp.

EEG Epoch Processing

For each trial:

  • EEG window is extracted from:
    • −200 ms to +800 ms relative to stimulus

Preprocessing:

  • bandpass filtering (0.1–30 Hz)
  • baseline correction using pre-stimulus window

Feature Extraction

We compute features from the P300 window (300–600 ms):

  • peak amplitude
  • mean amplitude
  • latency of peak
  • area under the curve (AUC)
  • composite P300 salience score
  • channel contribution (especially Pz, Cz, Fz)

We also compute bandpower features:

  • delta
  • theta
  • alpha
  • beta
  • gamma

Data Logging

Each trial is saved as a structured record:

  • image ID
  • image category
  • timestamp
  • EEG features
  • bandpower features
  • signal quality metrics
  • salience score
  • decision metadata

Importantly:

No data is discarded.
Noisy or incomplete signals are preserved with warnings.

Live UI

The EEG interface displays:

  • real-time Muse waveform
  • bandpower distribution
  • stimulus markers
  • salience signals

This makes the neural pipeline transparent and inspectable.

2. Drone + Computer Vision System

The second subsystem processes real-world input.

Input Sources

  • live drone feed
  • shared screen
  • prerecorded footage
  • simulated thermal frames

Frame Processing

  • RSVP-style sampling of frames
  • alignment with EEG timestamps
  • event tagging based on neural signals

Computer Vision Pipeline

We implemented a modular CV system:

Object Detection

  • YOLOv8 for:
    • person detection
    • object detection

Fire and Smoke Detection

  • OpenCV:
    • HSV thresholding for fire
    • connected component analysis
    • motion clustering for smoke

Event Generation

The CV system produces:

  • victim candidates (clustered detections)
  • fire regions
  • smoke propagation patterns
  • risk zones

Temporal + Spatial Analysis

  • event timelines
  • detection clustering
  • heatmaps of activity
  • prioritized frame extraction

3. Fusion Layer (Key Innovation)

This is the core of PyroSense.

We fuse:

Machine signal:

  • CV detections
  • object confidence scores

Human signal:

  • EEG salience
  • attention spikes
  • temporal alignment

Result

Each frame receives a composite salience score.

This allows the system to:

  • elevate frames where humans subconsciously noticed something
  • suppress irrelevant detections
  • recover edge cases missed by CV

4. Firefighter Decision Interface

The final output is a structured operational dashboard.

Features:

  • prioritized “important frames”
  • detected victims
  • hazard zones
  • timeline of events
  • spatial heatmaps
  • AI-generated rescue summary

Output Example

The system produces:

  • likely victim locations
  • hazard progression timeline
  • recommended entry path
  • risk zones to avoid

What Makes PyroSense Different

1. The human becomes a sensor

We capture subconscious detection, not explicit input.

2. CV is guided, not replaced

We reduce noise and increase precision.

3. Real-time operation

This is a live system, not post-hoc analysis.

4. Multimodal fusion

We align brain signals, vision, and time into a single decision layer.

Impact

PyroSense improves:

  • Detection recall
  • Time-to-identification
  • Operator efficiency
  • Situational awareness

Most importantly:

It reduces missed detections in the exact scenarios where mistakes cost lives.

Demo Design

To ensure reliability:

  • CV drives baseline classification
  • EEG provides a real, visible salience signal
  • EEG influence is blended into the final score

This ensures:

  • stable performance
  • technical honesty
  • reproducibility

Future Work

  • improved EEG classifiers (per-user adaptation)
  • integration with eye tracking for spatial localization
  • deployment on real drone hardware
  • multi-operator signal fusion
  • training on expert firefighter datasets
  • reinforcement learning for adaptive thresholds

Ultimately

We are going to save lives, lives that a firefighter subconsciously recognized, but could never act on. Until now.

Built With

Share this project:

Updates