CrowdSense

Inspiration

Crowd crush incidents at concerts, festivals, and public events have spiked post-pandemic. Crowd crush incidents, such as the Astroworld concert incident in 2021, as well as the Itaewon Halloween crowd crush in 2022, both had warning signs visible on security camera footage, but when the emergency was called, many had already been injured or harmed. In these incidents security was present but the crowd density increased rapidly causing the incidents. Every public event with large crowds risks crowd crushes and requires security personnel to be watching the crowd and keeping track of overflow and capacity. As a result, I created a lightweight dashboard tool that watches the security feed as well as live videos of crowds to provide security live updates.

What it does

CrowdSense is a browser-based crowd risk dashboard that works off a webcam or an uploaded video clip:

• Tracks four different risk signals which include headcount, density, motion energy, and 10-second crowd surges, and combines these factors into a 0-100 risk score, weighted to give an emphasis on density and motion
• Detects people in the frame in real time using a tiled scanning of the frame in overlapping zoomed sections to catch people that are further away or covered by other people
• Has an aerial mode The aerial mode estimates headcount by measuring how much of the frame is "crowd-textured" versus open ground, calibrated against an on-screen reference circle, the same area-based approach used in real safety guidance for very dense crowds.
• Speaks alerts out loud using Deepgram's Aura voice agent, so any security or user utilizing CrowdSense attention is immediately directed to the urgency of the situation for quick preventative action.

How we built it

• Frontend: HTML/CSS/JS, TensorFlow.js running COCO-SSD client-side for person detection, Chart.js for the live risk timeline, and a custom canvas-based pixel-variance algorithm for the aerial estimator.
• Backend: a small Node.js/Express server that proxies Deepgram's /v1/speak endpoint for voice alerts, so the API key never has to live in browser JavaScript.
• No video or images ever leave the browser. Only short alert text is sent to Deepgram for speech synthesis.

Challenges we ran into

The biggest challenge was that the first version of the dashboard struggled to identify a correct headcount and kept reporting "1 person" on a frame, although there were several hundred, and completely failed on real aerial/drone crowd footage. Debugging that meant pulling the actual frames from our test video and realizing the failure was because the object detector was trained on eye-level photos and wasn't seeing people's faces above. We fixed this by

Accomplishments we're proud of

• Built and validated a custom area crowd-density estimator from scratch, and verified it against real aerial footage before shipping it.
• Got a full real-time pipeline working client-side entirely with detection, density, motion, and surge, which feeds a live risk score and voice alerts.

What we learned

• General-purpose object detectors carry strong assumptions about camera angle that aren't obvious until you test them outside that assumption, for example, aerial drone shots.
• Voice alerts change how a monitoring tool actually gets used. A number changing on a screen is easy to miss, while a voice alert helps grab your attention immediately, along with a risk score.
• Weighting crowd risk by density and other factors is important to efficiently identify the risk of crowd crush while also not underestimating or overestimating the numbers.

What's next for CrowdSense

• Integrate an AI agent so security can ask for situational assessments and recommended action on demand, instead of just updates on a dashboard.
• Swap the area-based aerial estimator for a proper crowd-counting density mode for real per-frame head counts instead of an area estimate.
• Multiple-angle video support with a single risk dashboard across a whole venue, so an operator can see which entrance or section is vulnerable to the risk of crowd crush.
• On-device deployment for venues without reliable internet, since the detection pipeline already runs fully client-side.

Built With

• chart.js
• coco-ssd
• css
• css3
• deepgram
• deepgram-api
• express.js
• html5
• javascript
• node.js
• tensorflow.js