CrashSense AI
Inspiration
Traffic accidents can happen in seconds, but emergency response often depends on someone noticing the incident, understanding what happened, identifying the location, and contacting the right responders.
Many existing computer vision systems focus only on detecting whether an accident has occurred. We wanted to go one step further.
CrashSense AI was inspired by a simple question:
What if a CCTV system could not only detect a crash, but also understand what happened before the collision and turn that visual evidence into useful emergency information?
Our goal is to build an intelligent traffic monitoring system that can detect vehicle collisions, analyze vehicle behavior before impact, estimate the likely contributing factors, and automatically create an actionable incident report.
For example, instead of only reporting:
"Accident detected."
CrashSense AI aims to provide context such as:
"Vehicle A was travelling straight through the intersection while the traffic signal was green. Vehicle B entered the intersection from the right during a red signal and crossed Vehicle A's trajectory before a T-bone collision occurred."
This additional context can help emergency operators understand an incident faster and make better decisions.
What it does
CrashSense AI analyzes traffic surveillance video using computer vision and temporal reasoning.
The system is designed to perform the following workflow:
- Detect vehicles in CCTV footage.
- Track each vehicle across multiple video frames.
- Analyze vehicle trajectories and movement patterns.
- Detect possible collisions.
- Identify the type of collision.
- Analyze events leading up to the crash.
- Detect relevant traffic context, such as traffic-light states where available.
- Generate an accident timeline.
- Estimate the severity of the incident.
- Generate an AI-assisted incident summary.
- Attach the CCTV location and GPS coordinates.
- Create an incident that can be sent to an emergency-response dashboard.
Instead of treating each video frame independently, CrashSense AI analyzes how vehicles move over time.
This allows the system to reason about events such as:
- vehicles crossing each other's trajectories,
- sudden braking,
- abnormal changes in direction,
- vehicles entering an intersection,
- red-light violations,
- rear-end collisions,
- T-bone collisions,
- head-on collisions,
- and other abnormal vehicle interactions.
How we built it
CrashSense AI combines multiple computer vision and AI components into a perception-decision-action pipeline.
Vehicle Detection
We use object detection to identify vehicles such as:
- cars,
- motorcycles,
- buses,
- and trucks.
OpenCV 5 is used as a core component of the video-processing pipeline.
Multi-Object Tracking
After detecting vehicles, the system assigns each vehicle a persistent ID and tracks it across frames using a multi-object tracking approach such as ByteTrack.
For every tracked vehicle, the system can extract information including:
- bounding box,
- center position,
- movement direction,
- trajectory,
- estimated velocity,
- acceleration,
- and relative distance to nearby vehicles.
Collision Detection
Rather than deciding that a collision occurred from a single image, CrashSense AI analyzes several consecutive frames.
Possible collision evidence includes:
- rapidly decreasing distance between vehicles,
- intersecting trajectories,
- sudden deceleration,
- sudden changes in vehicle orientation,
- abnormal vehicle displacement,
- and vehicles stopping immediately after contact.
Accident Classification
The system is designed to classify common accident types such as:
- T-bone,
- rear-end,
- head-on,
- sideswipe,
- and single-vehicle accidents.
Accident Timeline
One of the key features of CrashSense AI is its ability to reconstruct the moments leading up to an accident.
For example:
- -3.0 seconds: Vehicle A approaches the intersection.
- -2.4 seconds: Vehicle B approaches from the right.
- -1.6 seconds: Vehicle B enters the intersection.
- -0.8 seconds: Vehicle A begins emergency braking.
- 0.0 seconds: Collision detected.
- +1.2 seconds: Both vehicles stop.
This timeline provides structured evidence for the later reasoning stage.
Cause Analysis
CrashSense AI analyzes the event timeline to estimate likely contributing factors.
Examples include:
- red-light violation,
- unsafe intersection crossing,
- sudden lane change,
- vehicle cut-in,
- failure to stop,
- wrong-way driving,
- sudden braking,
- or loss of vehicle control.
The system intentionally uses terms such as "likely contributing factor" instead of declaring legal responsibility.
This is important because CCTV footage may not contain all the evidence required to determine fault under traffic law.
AI Incident Summary
After collecting structured evidence from the video pipeline, the system can generate a human-readable incident summary.
For example:
Vehicle A travelled straight through the intersection while the traffic signal was green. Vehicle B entered the intersection from the right during a red signal and crossed Vehicle A's trajectory. A T-bone collision occurred. The likely contributing factor was a red-light violation by Vehicle B.
The language model is not expected to guess what happened directly from the video.
Instead, it receives structured evidence generated by the computer vision pipeline, reducing the risk of unsupported explanations.
Agentic Vision
CrashSense AI follows a perception → reasoning → decision → action architecture.
Perception
OpenCV 5 and the vision pipeline analyze the video and extract evidence.
Reasoning
The system determines:
- whether an accident occurred,
- what type of accident occurred,
- what happened before the collision,
- and how severe the incident may be.
Decision
Based on confidence and severity, the system can decide whether to:
- ignore the event,
- request human verification,
- or create an emergency incident.
Action
If an incident is confirmed, the system can:
- create an incident record,
- attach CCTV evidence,
- include GPS coordinates,
- generate an accident summary,
- and send the event to an emergency-response dashboard.
This means the visual analysis directly changes what the system does next rather than simply producing a static computer vision result.
AWS Architecture
CrashSense AI is designed to run meaningful components on AWS.
A potential deployment architecture includes:
CCTV / Video Input
→ OpenCV 5 Processing
→ Vehicle Detection and Tracking
→ Accident Analysis
→ Backend API on AWS
→ Incident Storage
→ Emergency Dashboard
AWS services may include:
- Amazon EC2 for AI inference and backend services,
- Amazon S3 for storing accident video clips and evidence,
- Amazon CloudWatch for logs and system monitoring,
- and additional AWS services where appropriate for deployment and scaling.
The architecture is designed so that video analysis can eventually scale from a single CCTV stream to multiple camera locations.
Dataset
For development and evaluation, we are using the ACCIDENT benchmark dataset, which contains traffic surveillance videos involving different types of vehicle accidents.
The dataset provides useful information for tasks such as:
- accident detection,
- collision timing,
- accident classification,
- vehicle tracking,
- and trajectory analysis.
Traffic-light reasoning may require additional training data or manually annotated scenarios because traffic-light visibility varies significantly between CCTV installations.
What we learned
Building CrashSense AI taught us that accident detection is fundamentally different from simple object detection.
Detecting a car is relatively straightforward.
Understanding an accident requires reasoning over time.
The system must understand how multiple vehicles interact across several seconds of video, not just what appears in a single frame.
We also learned that the most important part of an AI safety system is not only prediction accuracy, but also evidence and explainability.
If the system reports that a vehicle entered an intersection during a red signal, it should be able to show the tracked vehicle, traffic-light state, trajectory, and relevant video frames that led to that conclusion.
Another important lesson was that generative AI should not replace the computer vision pipeline.
Instead, structured visual evidence should be generated first, and a language model should only be used afterward to transform that evidence into a clear incident report.
This hybrid architecture makes the system easier to evaluate and more trustworthy.
Challenges we faced
Temporal reasoning
The biggest challenge is determining what happened over time.
A collision cannot always be identified correctly from one frame, so the system must track vehicles before, during, and after the event.
Vehicle occlusion
Vehicles may temporarily disappear behind other vehicles, buildings, trees, or infrastructure.
This can affect tracking and trajectory reconstruction.
Perspective distortion
CCTV cameras can be installed at different heights and angles.
Because of perspective, movement in pixel coordinates does not directly correspond to real-world speed or distance.
Traffic-light association
An intersection may contain several traffic lights for different directions.
Simply detecting a red light is not enough.
The system must determine which traffic light applies to which vehicle.
False positives
Vehicles stopping close together, crossing at different depths, or becoming visually overlapped can sometimes appear similar to a collision.
We therefore use multi-frame evidence instead of a single-frame decision.
Cause analysis
Determining why an accident occurred is significantly more difficult than detecting the accident itself.
We address this by producing a structured timeline and evidence before generating a likely cause.
Responsible use
CrashSense AI should not automatically determine legal fault.
The system is designed as a decision-support and emergency-response tool.
For uncertain situations, it should request human verification instead of making unsupported conclusions.
Evaluation
We plan to evaluate CrashSense AI using both model accuracy and real-world system performance.
Possible metrics include:
Accident Detection
- Precision
- Recall
- F1-score
- False-positive rate
Accident Classification
- Accuracy
- Macro F1-score
System Performance
- Processing FPS
- Detection latency
- Time from collision to incident creation
Cause Analysis
For videos with known scenarios, we can compare the predicted contributing factor with the ground-truth event description.
We also plan to explicitly document failure cases such as:
- poor lighting,
- vehicle occlusion,
- unclear traffic lights,
- distant accidents,
- heavy rain,
- and complex multi-vehicle scenes.
Responsible AI
CrashSense AI is intended to support emergency operators and traffic-safety systems, not replace human judgment.
Accident explanations are presented as likely contributing factors rather than definitive legal conclusions.
When confidence is low, the system can request human review.
Important visual evidence, including relevant frames, vehicle tracks, and event timelines, can be preserved so that users can understand why the AI made a particular decision.
What's next
Our next goal is to improve CrashSense AI beyond basic accident detection.
Future work includes:
- near-miss detection,
- multi-camera accident tracking,
- real-time CCTV streaming,
- accident hotspot analysis,
- traffic-risk heatmaps,
- automated responder routing,
- and larger-scale deployment across smart-city camera networks.
Our long-term vision is to transform existing CCTV infrastructure from passive recording systems into intelligent safety infrastructure capable of detecting, understanding, and responding to dangerous traffic events.
Why CrashSense AI matters
Most traffic cameras record what happened.
CrashSense AI aims to understand what happened and help trigger the next action.
See the crash. Understand the cause. Act faster.
Built With
- 5
- agentic
- ai
- amazon
- amazon-web-services
- analysis
- bytetrack
- computer
- detection
- ec2
- fastapi
- multi-object
- object
- opencv
- python
- pytorch
- s3
- tracking
- video
- vision
- yolo