A01 Synergistic Self-learning Crowd Management System

Crowd Anomaly Detection
Crowd Flow Density
Android App User Feedback
Android App Crowd Information

Inspiration

Crowd management in the Al-Mashaer area during Hajj season receives special attention from the authorities every year and significant workforce is dedicated for this sole purpose. With the use of smartphones becoming ubiquitous every passing moment and the availability of an existing video surveillance infrastructure, it seems prudent to share the roles pilgrims and authorities have so far played individually and cooperate! Specifically, let the pilgrims become aware of real-time concise congestion information on their screens from the central command while at the same time voluntarily sending feedback based on the ground truth from their perspective using the same platform i.e smartphones.

What it does

It gives the pilgrims access to real-time crowd density information in the Al-Mashaer area to help them decide their course and/or plan their trip to Jamaraat at a specific time. This information is displayed as a heat-map laid out on a map of the region on their smartphones. The central command generates this information from the video streams using image analytics to estimate the pedestrian/crowd flow at various points. Thus, extensive computer vision/machine learning operations are required to generate robust crowd flow information. Additionally, it requires constant learning from the generated data to improve its performance with time. For this latter task, feedback from people on ground is extremely important. Thus, pilgrims can voluntarily send feedback related to the actual crowd density at a specific point which they can see and analyze with their own eyes. Other sources of feedback can be volunteers assigned duties at specific checkpoints and the staff at the control center looking at the video streams. Thus, crowd density is mainly estimated from video streams using computer vision/machine learning techniques and fine-tuned using human feedback. Additionally, the machine learning task is retrained periodically to make use of the human feedback. The application requires internet access to get this data from the central command. To facilitate operations, telecom operators can be taken on-board for special packages. The same crowd density data can be used to predict anomalies such as stampedes and local collisions.

How we built it

The video analysis requires optical flow estimation as the main tool. Accumulation of people in a specific region can then be estimated using flow equations. Subsequently, block-based techniques can be used to train an SVM model to detect anomalies. We are relying heavily on open-source computer vision libraries, OpenCV/Matlab CV Toolbox, for worker tasks and SVM training. The smartphone app will be built using Android Studio while the central command for crowd flow estimation will be currently based on Matlab for quick development. Eventually, this will be moved to a more stable and portable environment. Google maps Api has been incorporated to provide congestion information to users in the form of layout heat maps.

Challenges we ran into

1- selecting an appropriate training dataset for pedestrian flow svm training model. 2- Annotation of different regions in image sequences according to their flow. All regions were hand-labelled. 3- Choice of hyperparameters of svm model. 4- Choice of communication protocol between control and nodes. Vector data transmission vs. image data was considered for pros and cons.