Dash-cam video detection

Inspiration

This project aims to address issues of road safety and understanding driver faults by providing a comprehensive driver safety score based on real-time dashcam footage analysis. By leveraging computer vision and deep learning, this driver safety score can help judge the risk levels of drivers and their driving habits and enable insurance companies to better assess risk.

Our progress

While the project isn't just a proof on concept - since it has fully functional components - we primarily focused on developing the backend and the processing pipeline for the dashcam footage. Our main emphasis changed halfway through the project to extracting meaningful driving data, and calculating the driver score, rather than building a fully polished front-end interface. The core processing modules are working and form the foundation for further developments as they have many different use cases.

What it does

The system processes dashcam footage to extract driving-related features which can be used to measure a drivers safety, road behavior and risk level.

Key Features Extracted from Footage

Vehicle Speed: Estimated using optical flow between consecutive frames.
Following Distance: Calculates the distance between the recording vehicle and those ahead to detect tailgating.
Traffic Signs and Lights: Detects stop signs, traffic lights, and their current state using image segmentation.
Nearby Pedestrians and Hazards: Identifies pedestrians and other potential road hazards to assess situational awareness

How we built it

Optical Flow: Used PyTorch to run an optical flow library to calculate vehicle speed from frame sequences
Image Segmentation: Leveraged Detectron2 to detect and segment road signs, traffic lights, pedestrians, and other vehicles.
Backend: Flask API handles video uploads and processing the footage

Challenges we ran into

Bugs and Debugging

We encountered unexpected bugs, especially when attempting to integrate multiple complex models (optical flow, segmentation, and prediction). Of course debugging under time constraints proved to be challenging but the main issues stemmed from many attempts to optimize the code and models that we used.

Hardware limitation

Running computationally intensive models on large video files required optimization, especially when using MPS instead of CUDA on Apple Silicon devices as this was the most powerful machine we had access too. An oversight of hardware capabilities meant that we would not have enough time to train our model to further optimize it as well.

Segmentation Accuracy

Differentiating between similar-looking signs or detecting partially obscured objects was challenging

Optical Flow Libraries

Making sure the Optical Flow library worked for our use case and was compatible with our hardware was a huge struggle during this project

Dataset Collection

Obtaining diverse and annotated dashcam footage for training was not possible as we did not have the time to annotate ourselves and had to use purely use image recognition to collect data.

Accomplishments that we're proud of

We successfully integrated optical flow and segmentation models to extract meaningful data from videos.
Developed a working prototype capable of calculating driver scores from uploaded footage.
Created an extensible pipeline that can be improved with additional detection features.
Our perseverance and dedicate to our idea and staying motivate even though we were unable to achieve our initial goal

What we learned

We learned an insane amount of about video processing techniques and optical flow analysis
Learned how to optimize deep learning models for MPS devices and handle large video data efficiently.
Gain knowledge in the intricacies of driver behavior analysis and its practical applications in insurance and safety industries.

What's next for Dash-cam Video Detection

A Driver Score

An algorithm that calculates a score for a particular driver considering all the data extracted by our model. This was something we planned to do but was left as we prioritised our model.

Enhanced Detection