IGNIS

A QNX-powered fire detection system that uses computer vision to spot fire and smoke, shows live incident context, and uses an AI voice agent to make real-world lifesaving decisions.

Comment

What it does

IGNIS is a real-time fire and smoke detection system running on a Raspberry Pi 5 with QNX. The Raspberry Pi camera continuously captures the monitored area, and the Pi runs a local computer vision model to detect visual signs of fire or smoke.

The Pi sends authenticated camera frames, detections, confidence values, and incident state updates to a laptop backend. The dashboard shows the live camera feed, detection boxes, confidence, system health, incident history, and state transitions in real time.

The dashboard includes an Armed / Unarmed control. When IGNIS is unarmed, the camera feed and model detections still display, but the backend ignores new incident escalation events and blocks ElevenLabs voice/call requests. When IGNIS is armed, confirmed fire or smoke incidents can trigger the voice and demo dispatch workflow.

For our demonstration, IGNIS can use ElevenLabs to call an approved team phone number. That number represents a demo dispatch contact. It does not call 911 or any real emergency service.

IGNIS also drives a 3.5-inch Raspberry Pi LCD. The screen normally shows the IGNIS logo, and during confirmed fire alerts it flashes between black and red.

How we built it

IGNIS is divided into three connected layers:

  • Edge detection on QNX: The Raspberry Pi 5 runs QNX 8.0, captures camera frames through the QNX camera path, preprocesses frames with OpenCV, and runs a local fire/smoke TFLite model. Fire and smoke decisions remain local to the Pi.

  • Backend and real-time communication: A FastAPI backend on the laptop receives authenticated TCP traffic from the Pi, stores incidents in SQLite, manages incident state, handles arming, and sends live updates to the dashboard through WebSockets.

  • Dashboard and communication: A Vite React + TypeScript dashboard displays the live camera stream, detections, confidence, Pi connection state, incident history, system health, armed/unarmed state, and ElevenLabs voice controls. When the system is armed, the backend allows ElevenLabs voice sessions and approved demo dispatch calls.

Our stack includes QNX, Raspberry Pi 5, Raspberry Pi Camera, Python, OpenCV, TensorFlow Lite runtime, FastAPI, authenticated TCP, WebSockets, React, TypeScript, Vite, SQLite, ElevenLabs, and a 3.5-inch Raspberry Pi LCD.

We collaborated through a shared GitHub repository and developed locally using VS Code, Cursor, Claude Code, and Codex. We accessed the Raspberry Pi through SSH to compile, test, and debug the QNX hardware, camera, LCD, and computer vision pipeline.

Challenges we ran into

The biggest challenge was getting the Raspberry Pi, camera, model runtime, LCD, and QNX environment working together. QNX exposes hardware differently from a typical Linux Raspberry Pi setup, so camera access, SPI/LCD control, dependencies, networking, and process management required a lot of debugging.

The model runtime also required care. Ordinary Linux ARM TensorFlow Lite wheels are not compatible with QNX, so the project needed a QNX-compatible Python TFLite runtime and a model artifact with a matching manifest.

Integrating ElevenLabs was another challenge. The voice agent needed the right incident context, the backend had to enforce demo-call safety rules, and the dashboard needed to make it clear when the system was armed or unarmed.

Seeing the full pipeline work, from visual fire/smoke detection on the Pi to live dashboard updates and an approved demo phone call, made that debugging worthwhile.

Accomplishments that we're proud of

We are proud that we built an end-to-end QNX Raspberry Pi fire/smoke detection system.

IGNIS does more than show a computer vision prediction. It connects embedded visual detection to a real-time dashboard, incident state management, local LCD alerting, and an AI voice/demo dispatch workflow.

The working loop is:

  1. The Raspberry Pi camera observes the scene.
  2. The QNX Pi runs local fire/smoke inference.
  3. The Pi sends frames, detections, and state updates over authenticated TCP.
  4. The laptop backend receives and stores the incident.
  5. The dashboard updates in real time.
  6. When armed, ElevenLabs voice and approved demo dispatch workflows can run.
  7. The local LCD shows the IGNIS logo normally and flashes during confirmed alerts.

Bringing QNX, embedded hardware, computer vision, full-stack development, local display control, and conversational AI into one working system was the highlight of the project.

What we learned

We learned how to deploy and debug a camera-based computer vision pipeline on a Raspberry Pi running QNX. This included camera capture, OpenCV preprocessing, TFLite inference, authenticated networking, and the constraints of an embedded real-time operating system.

We also learned how to connect a FastAPI backend to a React dashboard through WebSockets for real-time updates. Through ElevenLabs, we gained experience building a voice workflow that receives live incident context and can place approved outbound demo calls.

Most importantly, we learned how to coordinate several independently complex systems into one reliable workflow.

What's next for IGNIS

Our next step is to make IGNIS more reliable, informative, and suitable for real-world environments. We plan to:

  • Add smoke, heat, and physical fire-sensor inputs alongside the camera
  • Improve detection accuracy and reduce false alarms
  • Improve zone configuration for areas such as kitchens, outlets, and battery-charging areas
  • Preserve incident footage and generate evidence-based event timelines
  • Add stronger local alarms that continue working if the network is unavailable
  • Improve occupant verification and escalation workflows
  • Add more robust startup/watchdog behavior on the Pi
  • Integrate only with authorized monitoring or emergency-response systems where legally and operationally appropriate

Our long-term goal is for IGNIS to become an intelligent layer between fire detection and emergency response. Instead of only sounding an alarm, it can observe the situation, communicate what is happening, and help responders receive useful information sooner.

Built With

Share this project:

Updates