About the Project

Inspiration

Urban intersections are among the most complex and dangerous parts of a transportation system. Vehicles, cyclists, and pedestrians often move through the same space with limited visibility and incomplete information. Even when traffic cameras are available, they are usually used only for recording or manual monitoring rather than real-time risk prediction.

This inspired us to build CrossSafe, an AI-powered intersection safety system that predicts short-term motion and identifies potential conflicts before they happen.

Our goal was not simply to detect objects in a video. We wanted to understand how each road user was moving, estimate where they were likely to go next, and generate an early warning when two predicted paths were likely to overlap.

The main idea can be summarized as:

[ \text{Perception} + \text{Motion Prediction} +

\text{Risk Analysis}

\text{Proactive Safety} ]

Instead of reacting after a dangerous event occurs, CrossSafe attempts to recognize risk several seconds in advance.

What It Does

CrossSafe processes traffic video from an intersection and detects vehicles, cyclists, and pedestrians. It then tracks each object across frames, estimates its motion state, predicts its future trajectory, and calculates the probability of a collision or near-miss.

The system provides:

• Real-time road-user detection and tracking;
• Short-term trajectory prediction;
• Collision-risk estimation;
• Visual warnings for high-risk interactions;
• A dashboard showing active agents and risk levels;
• Event logging for later analysis.

For each detected agent, the system estimates a state vector:

[ \mathbf{s}i^t = \left[ x_i^t, y_i^t, v{x,i}^t, v_{y,i}^t, a_{x,i}^t, a_{y,i}^t \right], ]

where (x_i^t) and (y_i^t) represent position, (v_{x,i}^t) and (v_{y,i}^t) represent velocity, and (a_{x,i}^t) and (a_{y,i}^t) represent acceleration.

Using the recent motion history, the prediction module generates a future trajectory:

[

\hat{\mathbf{Y}}_i

\left[ \hat{\mathbf{p}}_i^{t+1}, \hat{\mathbf{p}}_i^{t+2}, \dots, \hat{\mathbf{p}}_i^{t+T} \right]. ]

The risk module compares predicted trajectories between road users. If two trajectories approach each other within a small spatial and temporal margin, the interaction is marked as potentially dangerous.

How We Built It

We divided CrossSafe into four main modules: perception, tracking, trajectory prediction, and risk visualization.

Perception

We used a pretrained object-detection model to identify vehicles, buses, bicycles, motorcycles, and pedestrians in each video frame.

Each detection contains:

• Object category;
• Bounding-box coordinates;
• Confidence score;
• Frame timestamp.

To reduce unstable predictions, detections below a confidence threshold were removed. We also applied non-maximum suppression to eliminate duplicate bounding boxes.

Multi-Object Tracking

Object detection alone does not preserve identity across time. We therefore integrated a multi-object tracker that assigns a persistent identifier to each road user.

The tracker associates detections between consecutive frames using a combination of motion consistency and visual similarity.

A simplified association cost is:

[

C_{ij}

\lambda_d D_{ij} + \lambda_a A_{ij}, ]

where (D_{ij}) is the spatial distance between a tracked object and a new detection, (A_{ij}) is the appearance difference, and (\lambda_d) and (\lambda_a) control their relative importance.

The resulting tracking history provides a sequence of positions for every active agent.

Coordinate Transformation

Pixel coordinates are not sufficient for meaningful risk estimation because image distances depend on perspective. We therefore used a planar homography to map points from the image plane into an approximate ground-plane coordinate system.

The transformation is defined as:

[ \begin{bmatrix} X \ Y \ 1 \end{bmatrix} \sim \mathbf{H} \begin{bmatrix} u \ v \ 1 \end{bmatrix}, ]

where ((u,v)) is an image coordinate, ((X,Y)) is the corresponding ground-plane coordinate, and (\mathbf{H}) is the homography matrix.

This allowed us to estimate distances and motion in meters rather than pixels.

Trajectory Prediction

For the hackathon prototype, we implemented a lightweight sequence-prediction model. The model uses the recent positions and velocities of an agent to predict its movement over the next few seconds.

The prediction model takes a historical sequence:

[ \mathbf{X}_i = \left[ \mathbf{p}_i^{t-H+1}, \dots, \mathbf{p}_i^t \right] ]

and produces:

[ \hat{\mathbf{Y}}i = f{\theta}(\mathbf{X}_i). ]

The model was trained by minimizing the average displacement error:

[

\mathcal{L}_{\mathrm{ADE}}

\frac{1}{NT} \sum_{i=1}^{N} \sum_{\tau=1}^{T} \left|

\hat{\mathbf{p}}_i^{t+\tau}

\mathbf{p}_i^{t+\tau} \right|_2. ]

To keep the system responsive, we used a compact architecture instead of a large model. This made it possible to run detection, tracking, prediction, and visualization on the same machine.

Risk Estimation

For each pair of agents, we calculated the minimum predicted distance:

[

d_{ij}^{\min}

\min_{\tau \in {1,\dots,T}} \left|

\hat{\mathbf{p}}_i^{t+\tau}

\hat{\mathbf{p}}_j^{t+\tau} \right|_2. ]

We also estimated the time at which this minimum distance occurred:

[

\tau_{ij}^{*}

\arg\min_{\tau} \left|

\hat{\mathbf{p}}_i^{t+\tau}

\hat{\mathbf{p}}_j^{t+\tau} \right|_2. ]

An interaction was considered high-risk when the predicted distance was small and the potential conflict was expected to occur soon.

We defined a simple risk score:

[

R_{ij}

\exp \left( -\frac{d_{ij}^{\min}}{\sigma_d} \right) \cdot \exp \left( -\frac{\tau_{ij}^{*}}{\sigma_t} \right), ]

where (\sigma_d) and (\sigma_t) control sensitivity to distance and time.

A higher value of (R_{ij}) indicates a more urgent interaction.

Dashboard and Visualization

We built a web dashboard to display:

• The live or uploaded traffic video;
• Detected road users;
• Historical trajectories;
• Predicted future trajectories;
• Pairwise risk warnings;
• A list of recent high-risk events.

Low-risk trajectories are shown normally, while dangerous interactions are highlighted with warning markers and labels.

The backend processes video frames and sends structured results to the frontend. The frontend then renders the detections and predictions on an interactive canvas.

Technologies Used

We used the following technologies:

• Python for the main backend;
• PyTorch for trajectory prediction;
• OpenCV for video processing and coordinate transformation;
• A pretrained object-detection model for road-user detection;
• A multi-object tracking algorithm for identity association;
• FastAPI for backend services;
• WebSockets for real-time communication;
• React for the frontend dashboard;
• Docker for environment packaging;
• GitHub for collaboration and version control.

Challenges We Faced

Maintaining Stable Object Identities

One of the first challenges was identity switching. When vehicles crossed each other or became partially occluded, the tracker occasionally assigned a new identifier to the same object.

This caused discontinuities in the trajectory history and reduced prediction quality.

We improved the result by combining spatial association with appearance information. We also introduced a short retention period so that an object was not immediately deleted when it disappeared for a few frames.

Converting Pixels Into Physical Distance

Risk analysis in pixel coordinates produced misleading results. Objects near the top of the image appeared closer together than objects near the camera, even when their real-world distances were similar.

We addressed this problem by calibrating the road plane with manually selected reference points. Although this approximation is not as accurate as full camera calibration, it was sufficient for the prototype.

Noisy Velocity Estimation

Directly differentiating consecutive positions produced unstable velocity values:

[

\mathbf{v}_i^t

\frac{ \mathbf{p}_i^t-\mathbf{p}_i^{t-1} }{ \Delta t }. ]

Small tracking errors resulted in large velocity fluctuations. We applied temporal smoothing to reduce this noise:

[

\tilde{\mathbf{v}}_i^t

\alpha \mathbf{v}_i^t + (1-\alpha)\tilde{\mathbf{v}}_i^{t-1}. ]

This improved both trajectory stability and risk estimation.

Balancing Accuracy and Speed

The complete pipeline contained several computationally expensive stages. Running detection, tracking, prediction, and visualization for every frame initially resulted in unacceptable latency.

We improved performance by:

• Processing video at a controlled frame rate;
• Reusing tracking states between detections;
• Batching model inference;
• Limiting trajectory prediction to active and relevant agents;
• Using a smaller prediction network;
• Avoiding unnecessary data transfers.

The main engineering trade-off was:

[

\text{System Quality}

\alpha \cdot \text{Accuracy} +

\beta \cdot \text{Responsiveness}

\gamma \cdot \text{Computational Cost}. ]

For the hackathon, we prioritized a stable real-time demonstration over maximum benchmark accuracy.

Limited Training Data

We did not have enough labeled intersection data to train every component from scratch. We therefore relied on pretrained perception models and used a small public motion dataset for initial trajectory training.

To adapt the prediction model to our demonstration videos, we normalized trajectories relative to each agent's current position and heading. This reduced sensitivity to camera location and road orientation.

What We Learned

This project taught us that building an intelligent transportation system requires more than training a single model. The quality of the final result depends on the entire pipeline.

We learned how to connect perception, tracking, coordinate transformation, prediction, and visualization into one functional application.

We also learned several practical lessons:

• Accurate tracking is essential for reliable trajectory prediction;
• Coordinate systems must be handled carefully;
• A theoretically strong model may not be suitable for real-time deployment;
• Data quality can matter more than model complexity;
• Visual explanations make AI predictions easier to understand;
• A useful prototype must remain stable under imperfect real-world inputs.

Most importantly, we learned to reduce a broad safety problem into a small set of testable components and improve each component independently.

Accomplishments

During the hackathon, we successfully built an end-to-end prototype that:

• Detects and tracks multiple road users;
• Maintains short-term motion histories;
• Predicts future movement;
• Identifies potentially dangerous interactions;
• Displays results through an interactive dashboard;
• Runs on prerecorded traffic video with near-real-time performance.

We are particularly proud that the system does not stop at object detection. It converts visual observations into motion predictions and interpretable safety warnings.

What Is Next

CrossSafe is currently a proof of concept. Several improvements are planned.

First, we would replace the lightweight trajectory model with an interaction-aware prediction architecture that considers nearby agents, road geometry, lane topology, and traffic signals.

Second, we would improve uncertainty estimation. Instead of predicting only one trajectory, the model could generate multiple possible futures:

[ p(\mathbf{Y}\mid\mathbf{X},\mathcal{M}), ]

where (\mathbf{X}) represents motion history and (\mathcal{M}) represents map and scene context.

Third, we would use full camera calibration or depth sensors to improve world-coordinate estimation.

Fourth, we would evaluate the system on a larger dataset using metrics such as:

[

\mathrm{ADE}

\frac{1}{T} \sum_{\tau=1}^{T} \left|

\hat{\mathbf{p}}^{t+\tau}

\mathbf{p}^{t+\tau} \right|_2, ]

and

[

\mathrm{FDE}

\left|

\hat{\mathbf{p}}^{t+T}

\mathbf{p}^{t+T} \right|_2. ]

Finally, we would develop a city-scale monitoring platform that aggregates risk events from multiple intersections. Such a system could help transportation agencies identify dangerous locations, evaluate traffic-control policies, and prioritize infrastructure improvements.

Our long-term vision is to transform existing traffic cameras from passive recording devices into proactive safety sensors.

Built With

• ai
• dl