Pedestrian Trajectory Analysis for Autonomous Vehicle Safety Inspiration The rapid advancement of autonomous vehicles promises a future of safer, more efficient transportation. However, the greatest challenge to this vision lies in navigating the unpredictable nature of human behavior, especially that of pedestrians. Inspired by the mission to make autonomous systems safer and more attuned to their environment, this project was born. It serves as a proof-of-concept for a system that moves beyond simple object detection towards a more intelligent understanding of pedestrian trajectory and intent, aiming to proactively identify and flag at-risk individuals before they step into danger.
How It Works: The Technical Pipeline The system processes video footage through a multi-stage pipeline, where each stage adds a layer of analysis. The ultimate goal is to identify a person, understand where they are and where they are going, and determine if their path intersects with a dangerous area, such as a road.
Step 1: Environment Mapping with BiSeNetV2
The first step in understanding risk is to define the "danger zone." Using the BiSeNetV2 road segmentation model, the system analyzes each video frame to create a binary mask, precisely identifying all drivable road surfaces. This mask serves as our ground truth for determining if a pedestrian is currently on the road or is likely to enter it.
Step 2: Human Pose Estimation with Detectron2
With the environment mapped, the focus shifts to the people within it. The powerful Detectron2 framework is used for keypoint detection. It processes each frame to identify humans and, critically, to extract the 2D coordinates of their major joints (shoulders, hips, knees, etc.). This skeletal data is the foundation for all subsequent motion analysis.
Step 3: Vectorization and Motion Tracking
Raw keypoint coordinates are just dots on a screen. To understand movement and orientation, we must convert them into meaningful vectors. For each detected person, a shoulder vector and a torso vector are calculated.
Let \(P_{\text{left-shoulder}}\) and \(P_{\text{right-shoulder}}\) be the coordinates of the shoulder keypoints. The shoulder vector
$$
\vec{S} = P_{\text{right-shoulder}} - P_{\text{left-shoulder}}
$$
be the midpoint of the hip keypoints and \(P_{\text{mid-shoulder}}\) be the midpoint of the shoulder keypoints. The torso vector
$$
\vec{T} = P_{\text{mid-shoulder}} - P_{\text{mid-hip}} = (T_x, T_y)
$$
To get a consistent directional vector \(\vec{D}\) that points orthogonally "out" from the person's chest, we compute the cross product of the torso vector \(\vec{T}\) and the shoulder vector \(\vec{S}\).
$$
\vec{D} = \vec{S} \times \vec{T} =
\left|
\begin{array}{ccc}
\mathbf{i} & \mathbf{j} & \mathbf{k} \
S_x & S_y & 0 \
T_x & T_y & 0
\end{array}
\right|
= (S_x T_y - S_y T_x)\mathbf{k}
$$
Finally, a simple distance-based tracking algorithm links individuals across consecutive frames, allowing us to build a history of their position and calculate motion dynamics.
Step 4: Velocity and Future-State Prediction
With a person's position tracked over time, we can calculate their velocity. The instantaneous speed is calculated as the displacement in pixels between frames, divided by the time elapsed. To ensure stability, a moving average of the speed over the last 10 frames is used. $$ \text{speed} = \frac{||\text{position}(t) - \text{position}(t-1)||}{\Delta t} $$ Using this smoothed velocity and the person's most recent direction vector, the system projects their position 1 second into the future. This prediction is key to the system's proactive safety flagging.
Step 5: Risk Assessment and Flagging
A person is deemed "at-risk" and flagged with a red bounding box if either of two conditions is met:
Their current position (defined by their hip-center) is inside the road mask.
Their predicted future position is inside the road mask.
This dual-condition approach ensures that the system flags not only people already in danger but also those who are on a clear trajectory to enter it.
The Learning Journey This project was a profound lesson in the power and complexity of model integration. The core achievement was not just using a pre-trained model but successfully fusing two distinct, highly specialized models—one for environmental segmentation (BiSeNetV2) and one for human pose estimation (Detectron2)—into a single, functional pipeline. This required managing complex software dependencies within the Google Colab environment, handling CUDA configurations, and ensuring the output of one model could be seamlessly processed as the input for the next. Along the way, we gained invaluable experience with the principles of pose detection, the challenges of working with real-world video datasets, and the project management skills needed to deliver a result under tight time constraints.
Challenges Faced Data Scarcity: Finding high-quality, public video datasets of pedestrians in varied urban environments proved to be a significant challenge. This made it difficult to test and refine the model's performance across different lighting conditions, crowd densities, and camera angles.
Technical Hurdles: The initial phase was marked by struggles with package dependencies. An early attempt to use mmpose failed due to environment conflicts, which necessitated a pivot to the more robust Detectron2 framework.
Model Sensitivity: Tuning the confidence thresholds for person detection was a delicate balancing act. A low threshold resulted in many "false positives" (e.g., boxes appearing on fire hydrants), while a high threshold risked failing to detect actual people, especially those far from the camera. The final implementation enforces a "no keypoints, no box" rule to mitigate this.
Time Constraints: As with any project, working within a limited timeframe forced a focus on core functionalities and pragmatic solutions over more complex, time-consuming alternatives.
Future Directions This proof-of-concept lays the groundwork for a much more sophisticated system. The next evolution would be to develop an advanced intent analysis model. This could involve analyzing more subtle cues—such as head pose (is the person looking at traffic?), foot orientation, and gait—to train a machine learning model that classifies pedestrian intent (e.g., "intends to cross," "waiting," "unaware"). The simple red box could evolve into a more nuanced flagging system with varying levels of alert. Ultimately, the vision is for this system to become a critical component of an autonomous vehicle's internal perception stack, providing a crucial layer of predictive safety.
Log in or sign up for Devpost to join the conversation.