Inspiration

Many pedestrian deaths happen at crossings without traffic lights, where the driver sees the person too late: they step out from behind a parked car, a stopping car hides them from the one overtaking it, or it is simply dark. In the EU almost one road death in five is a pedestrian. We asked a simple question: what if the crossing itself could see the pedestrian the driver cannot, and warn early enough to matter, without depending on the Internet?

What it does

Smart Crossing is a small node that mounts on an existing pole at an unsignalised crossing. An Arduino UNO Q with a camera:

detects pedestrians, cyclists and vehicles with an on-device model,
tracks each road user and estimates speed and direction,
decides with an explicit, tested rule whether drivers must be warned,
warns through lights driven by the board's STM32 microcontroller,
counts crossings by class and direction,
and does all of it offline: no server, no cloud, and images never have to leave the board.

The rule has four states. WARN (steady amber) means someone is waiting and a vehicle will reach the crosswalk within 6 s. ALERT (flashing) means someone is on the crosswalk and a vehicle is 4 s away or less. OFF means nothing is at risk. The core quantity is the time a vehicle needs to reach the crosswalk:

$$ t_{\text{reach}} = \frac{d}{v}, \qquad \text{WARN if } t_{\text{reach}} \le 6\,\text{s}, \quad \text{ALERT if } t_{\text{reach}} \le 4\,\text{s} $$

where $$d$$ is the vehicle's distance to the crosswalk and $$v$$ its speed.

A web dashboard shows the live camera, the light state, counts and board health. It also shows a second use case: a network of nodes at signalised junctions that talk to each other and to an ambulance over a LoRa-like radio, giving it a green wave and detours around blocked roads, simulated on real streets of Aalborg.

How we built it

Model. We fine-tuned YOLO26n from COCO weights on a 3-class dataset (pedestrian, cyclist, vehicle; 1,219 labelled images). We exported it at 288x384, matching the camera's 4:3 frame so no pixels are wasted on padding. Then we quantised it to INT8, keeping the detection head in FP32. On the UNO Q's four Cortex-A53 cores it runs in 88 ms, for 8.2 analysed frames per second live, with test mAP@50 of 0.628.
Pipeline. Python on the board's Linux side reads the IMX219 camera, runs the detector and a tracker, computes per-object facts (zone, motion, heading, time to reach the crosswalk) and applies the rule in about 7 ms per frame.
Firmware. The STM32 sketch drives the LEDs, buzzer and LCD and exposes them to Linux over the Bridge. A watchdog blinks blue if the vision side goes silent for 2 s.
Rule as code and as a document. rules.py and rules.md describe the same rule. Unit tests cover every scenario, and a scripted demo video checks 7 of 7 scenarios end to end.
Dashboard. React and three.js, with a 3D model of the board and live data from its status API.
City simulation. SUMO runs junction nodes, an ambulance unit and a simulated radio. We fed it real footage from the AAU RainSnow dataset by running our own detector on recorded Aalborg videos.

Challenges we ran into

A flickering camera. The IMX219's software auto exposure oscillated between black and white frames. We bypassed it by capturing RAW Bayer frames and doing our own debayering, white balance and, later, our own auto exposure.
Speed versus accuracy on a CPU. Every pixel counts at 8 FPS. Bigger inputs and bigger models were more accurate but too slow, so we picked the input size and quantisation scheme by measuring on the board, not on a laptop.
Small, distant pedestrians. Pedestrians are the hardest class (AP 0.461): at CCTV distances they are only a few pixels tall. We tuned per-class thresholds and tracking to avoid missing people, accepting extra amber warnings.
Safety semantics. Turning detections into a warning that is never silently wrong took several iterations. In the end the light never goes off while someone is on the crosswalk, it never says "all clear" when the camera cannot see, and it turns red only after a second confirmation.
[A 24-hour hackathon with four people pushing to the same repository.]

Accomplishments that we're proud of

A full detection, tracking and decision pipeline running live on a €[xx] board, with no cloud.
An explicit, readable rule: every light has a reason you can print and test.
Honest failure: the node reports FAULT instead of pretending everything is fine.
In simulation, on a 4-junction grid (240 runs), the node network brings the ambulance in 27.1 s sooner than fixed-time signals, with no collisions. On 14 real junctions in Aalborg, the green wave cuts its mean trip from 451.7 s to 253.4 s.

What we learned

On edge hardware, measure on the device. Laptop numbers lied to us more than once.
Quantisation is not free: keeping the detection head in FP32 was the difference between usable and unusable INT8.
A safety rule is a product decision before it is code: writing the ground rules first made every later threshold easier to choose.
Real data is messy. Weather, night and occlusions matter far more than benchmark scores suggest.

What's next for Smart Crossing

Retrain on frames from our own camera and at night to raise pedestrian recall.
Calibrate scenes in metres on site, and validate the thresholds with real traffic.
Mount the external warning light and test a node on a real pole.
An infrared illuminator or thermal camera for the dark case.
Move the node network from simulation to real radios, starting with an ambulance beacon on LoRa.