VigilEyes

Truckers drive 11+ hours daily in isolation. VigilEyes uses custom YOLOv8 and sensor fusion to detect fatigue and swerving, providing a real-time risk score to keep the backbone of our society safe.

Comment

Inspiration

Long-haul truckers are the invisible backbone of our global economy, yet they are among the most isolated workers in the world. Spending 11+ hours a day behind the wheel and over 300 days a year on the road, these individuals operate in a "functional vacuum" where the pressure to deliver outweighs the opportunity for self-care.

We realized that while modern logistics is obsessed with tracking the package, it rarely tracks the person.

Because of this disconnect, it has created quite a dangerous reality:

  • The Silent Struggle: The reason why it's a silent struggle because fatigue isn't a loud event. Fatigue is a series of "micro-sleeps" and slow reaction times that a driver might not even notice until it’s too late.

  • A World Not Built for Them: Navigating heavy machinery while exhausted is a massive mental burden, yet there are few tools designed to act as an empathetic co-pilot. Because the average person does not drive a semi-truck, the unique psychological and physical burdens of these drivers often go unnoticed and unaddressed.

  • The Goal: We wanted to build something that honors their contribution by looking out for their safety. Our goal is to ensure that the people responsible for delivering our world can safely return to theirs.

What it does

VigilEyes is an AI-powered "Digital Co-Pilot" designed to monitor and protect long-haul truckers from the dangers of fatigue and isolation. By turning a smartphone into an intelligent safety hub, the app actively watches over the driver to ensure they stay alert and safe during grueling shifts.

Key Features:

Dynamic Risk Score: The core of VigilEyes is a dynamic 0–100 Risk Score. The app constantly evaluates driver behavior to determine their level of exhaustion.

  • 0–59 (Safe/Alert): The driver is focused and showing normal physiological signs.
  • 60+ (At Risk): The app identifies early signs of fatigue. If the score stays in this range, it triggers an active alert to re-engage the driver.
  • 80–100 (Critical): Immediate intervention is required to prevent a potential accident.

Fatigue Detection: The app monitors the driver’s face in real-time to detect "micro-sleeps" (eyes closed for too long) and frequent yawning.

Driving Pattern Analysis: Beyond just watching the driver, VigilEyes tracks the vehicle’s movement. It detects erratic swerving and sudden, "jerky" braking which are quite common physical symptoms of a tired driver losing focus.

Intelligent Sensor Fusion: By combining visual cues with vehicle motion data, the system creates a high-fidelity safety profile. It knows the difference between a bump in the road and a driver drifting out of their lane.

Empathetic Alerting: When the risk score crosses the danger threshold, the app provides immediate audio-visual feedback to act as a supportive co-pilot, reminding the driver that it’s time to take a break.

How we built it

We engineered a distributed system designed to handle high-frequency data streams with minimal latency across four main layers:

The Mobile Sensor Hub (React Native): Our mobile app acts as the primary data collector. It captures high-precision telemetry from the accelerometer and gyroscope while polling video frames every 50ms.

Secure Real-Time Communication: Because safety alerts require immediate feedback, we utilized WebSockets via Tailscale to bridge the phone to our backend. This ensures a secure, low-latency tunnel for streaming sensor data and image frames directly to our processing engine.

The Intelligence Engine (Flask Backend): We developed a modular backend that processes incoming data through three distinct specialized models:

  • Vision Models: Two separate YOLO models—one custom-trained to detect eye closure duration and another dedicated to identifying yawning patterns.
  • Kinetic Model: A dedicated logic layer for processing Gyroscope and Accelerometer data to detect erratic driving behaviors like sudden swerving or hard braking.
  • Sensor Fusion: These independent risk factors (weighted 60% vision, 40% motion) are merged into our final 0–100 Risk Score and sent back to the phone instantly.

The Monitoring Dashboard (React Frontend): To provide a "dispatch view," we built a web-based dashboard that pulls real-time metrics from the backend via HTTPS, allowing us to visualize the driver's risk data as it happens.

Challenges we ran into

Pivoting Architecture from the Original Idea:

  • We originally planned to build everything to a mobile device using MediaPipe within React Native. However, we quickly discovered that the Expo Go camera would un-mount itself when sending all the stream data to the MediaPipe api and have latency issues on the mobile device.
  • Even after optimizing, we found that running heavy computer vision models locally made the phone overheat and caused significant frame drops. The models were simply too heavy for mobile hardware to process at the speed we required.
  • To solve the latency, unmounting and hardware issues, we had to completely redesign our system. We moved the logic of the app to a Flask backend, implementing a Tailscale WebSocket tunnel to stream frames every 50ms. This allowed us to offload the heavy lifting while keeping the feedback loop nearly instantaneous.

Inaccurate Pre-trained Models:

  • When we tested existing "drowsiness" models found online, the accuracy was poor, often failing to distinguish between a long blink and actual micro-sleep. This forced us to pivot and train our own YOLOv8 models from scratch to ensure the precision needed for a 0–100 risk score.

Integration & Merge Hell:

  • Once all the individual features were finished, merging them into a single repo led to massive merge conflicts. It took a while to resolve the versioning issues and get the sensor fusion logic working across the entire stack.

Accomplishments that we're proud of

  • We are incredibly proud of what we achieved in just 24 hours, especially considering that 8 hours into the hackathon, we weren't even sure if our MVP which was simply detecting eye closure on a phone was technically possible.
  • This was our team's first mobile application using React Native. Moving from a web app mindset to a more complex distributed architectural redesign midway through the evening was tough but very rewarding. It let us be able to work on many more features after.
  • This was also our first time training a model, it was incredibly exciting to move beyond pre-made APIs to curating a dataset of 4,000+ data points, generating our own custom YOLOv8 weights, and writing the scripts to integrate them into our backend.
  • This was our team's first time interfacing with gyroscope and accelerometer data, and making sense of it using a Random Forest Classifier after normalizing the data.
  • Our whole pipeline has extremely low latency, which is something we're especially proud of given the technical constraints we faced with latency. Our project had started with a camera capture every second, and ended with a capture every 50ms, resulting in a 95% reduction in latency. Additionally, utilizing a tailscale websocket instead of an https request for our image data cut our latency by ~75% alone. On top of the data hand-off, we were able to utilize YOLOv8 models to make near-instant image classifications and cut latency even further.

What we learned

  • We learned that while modern smartphones are able to gather critical and unique data, they aren't always built for continuous, high-fidelity computer vision. We discovered firsthand how on-device processing can lead to overheating and latency that compromises safety in a life-saving app.
  • We gained deep experience with WebSockets. We learned how to manage high-frequency data streams (every 15-50ms) and how to use Tailscale to create a secure, low-latency tunnel between a mobile device and a remote backend.
  • This was our first time working with React Native. We learned how to navigate the mobile lifecycle, specifically how to manage camera mounting and state when polling frames at high speeds without crashing the frontend.
  • We decided to train YOLOv8 models instead of utilizing an AI API for low latency, which is absolutely something we would do again in the future, when every millisecond counts for someone's safety.

What's next for VigilEyes

  • Real-Time Fleet Connectivity: Integrating GPS geolocation to allow drivers to see other "VigilEyes" users on their route. This would transform the road from a place of isolation into a connected community.

  • Proximity-Based Voice Chat: Implementing a "Digital CB Radio" feature where drivers can easily talk to nearby peers, sharing road conditions, safety warnings, or just having a conversation to stay alert and engaged.

  • Smart Rest-Stop Integration: Using the Risk Score to automatically suggest the nearest truck-friendly rest areas, fuel stations, or healthy food options when the system detects rising fatigue.

  • Predictive Analytics: Analyzing a driver's historical data to suggest optimal break times before the fatigue sets in.

  • Wearable Sensor Fusion: Incorporating data from smartwatches (like heart rate variability and blood oxygen levels) to create an even more precise 0–100 Risk Score.

DISCLAIMER: Our demo is for education purposes ONLY. Under no circumstances do we encourage dangerous driving, and in the video, the passenger was the one closing their eyes. Drive safe!!

+ 11 more
Share this project:

Updates