Environmental & Facility Auditor

Real-time industrial inspection platform powered by Gemini 2.0 Flash to detect leaks, environmental hazards, and energy waste with precision bounding-box tracking.

Comment

💡 Inspiration

Industrial facilities lose billions annually due to undetected pneumatic micro-leaks, thermal inefficiency, and safety non-compliance. Traditional manual audits are slow, expensive, and often dangerous. We built Environmental & Facility Auditor (formerly EcoAuditor) to prove that multimodal AI can transform static security/inspection video feeds into an automated, real-time diagnostic stream. By leveraging Gemini 2.0 Flash, we turn raw video frames into actionable engineering telemetry with zero manual overhead.

⚙️ What It Does

Environmental & Facility Auditor is an industrial inspection dashboard that processes live or uploaded video streams to detect physical anomalies in real time.

  • Deterministic AI Keyframe Inspection: Captures video frames client-side and streams them directly to Gemini 2.0 Flash for fast, structured JSON anomaly analysis.
  • Dynamic Visual Grounding: Overlays real-time, dynamic 2D bounding boxes ([ymin, xmin, ymax, xmax]) directly onto the video canvas, tracking localized defects frame-by-frame.
  • Playhead & Telemetry Synchronization: Intersecting timestamps lock findings directly to the video playhead. Clicking "Seek" on any audit card jumps the video directly to the precise second the anomaly occurs.
  • ROI & Impact Calculation: Automatically quantifies financial waste ($/yr) and energy loss (kWh) associated with detected leaks or mechanical strain.
  • Multi-Layer Inspection Controls: Toggles diagnostic thermal/acoustic overlays and toast feedback to give operators immediate visual acknowledgment of completed AI runs.

🛠️ How We Built It

  • Frontend Architecture: React (TypeScript) built with a unified, state-machine pattern to eliminate DOM desynchronization between HTML5 <video> elements and dynamic overlay canvases.
  • AI Engine: Integrated Gemini 2.0 Flash using structured JSON output schema to return precise normalized coordinates, severity tiers, and acoustic/visual telemetry cues.
  • Media Processing: Client-side HTML5 Canvas API for real-time keyframe extraction, bypassing CORS and server-side processing bottlenecks.

🚧 Challenges We Faced

  • DOM & State Synchronization: Keeping dynamic bounding box overlays accurately aligned with the HTML5 video playhead during seeking and tab-switching required refactoring our state machine to sync time updates continuously.
  • Eliminating AI Hallucinations: Early iterations suffered from generic anomaly prompts (e.g., flagging nonexistent electrical cables on simple pipe assemblies). We solved this by enforcing strict visual grounding rules in our Gemini system prompt, ensuring the model only reports verified pixel data.
  • Responsive Coordinate Normalization: Scaling raw API coordinate outputs [ymin, xmin, ymax, xmax] dynamically across varying video aspect ratios without offset errors.

📚 What We Learned

  • Gemini 2.0 Flash Speed: The speed of Gemini 2.0 Flash makes near-real-time visual telemetry for video analysis achievable directly in client web apps.
  • UX for AI Workflows: Immediate feedback cues—such as active toast notifications and strict loading indicators—are essential for maintaining user trust during AI extraction tasks.

🚀 What's Next for Environmental & Facility Auditor

  • Edge model deployment for low-latency offline factory processing.
  • Integration with IoT ultrasonic microphone arrays for multi-sensor acoustic heatmaps.
  • Automated maintenance work order dispatching via webhooks.

Built With

Share this project:

Updates

Submission history