Fly Marshal

Inspiration

At grassroots and club races, a car can spin at the far end of the circuit and race control won't see it for several seconds. Big championships solve this with hundreds of marshals and expensive fixed camera rigs, but smaller leagues can't afford either. We wanted to give them affordable, flexible, autonomous eyes in the sky, so their drivers get the same level of safety.

What it does

Fly Marshal is an AI-powered virtual marshal for motorsport. It watches the race through live telemetry, sends drones to trouble, and gives race control a clear picture of what's happening.

Incident detection: spots spins, offs, slides, contacts and stopped cars from live telemetry in about 0.36 seconds
Incident prediction: flags cars likely to have an incident in the next few seconds, so a drone can get into position early
Autonomous drone dispatch: the nearest drone flies to the scene and streams a live camera feed rendered inside the game
Vision model (VLM): runs in the background on drone footage to check for debris, smoke and blocked lines
Fruit-fly brain drone: one drone's heading is steered by a real 20,073-neuron FlyWire connectome, simulated live
Race-control dashboard: a 2D map, a full 3D circuit view, alerts, incident reports and drone cams you can pan, zoom and lock onto
Multi-track and sponsor-ready: rebuilds itself for whichever track is running, with custom car skins and sponsor boards

How we built it

Fly Marshal is a real-time pipeline from the simulator to race control.

Data: we recorded 96+ laps with 24 cars across 3 circuits (Vallelunga, Spa and Imola) in Assetto Corsa, logging every car 15 times a second
Machine learning: we engineered track-relative features, such as speed versus what's typical for that corner, extra yaw, unusual acceleration and time gaps. After comparing several model types, we chose a tuned LightGBM and logistic regression blend
Drone system: a dispatcher assigns drones to incidents, flown by PID or FlyBrain pilots, with a safety layer on top
FlyBrain: simulated photoreceptors in the connectome decide which way the drone turns, while conventional control holds speed and altitude
Computer vision: fresh frames from the in-game drone cameras go to a VLM, which adds what the camera sees to the incident report
Dashboard: a real-time web app built with Three.js and WebSockets, including a live visualization of the fly brain

Challenges we ran into

Generalization: our first model learned raw speeds and distances and flooded us with false alarms on a new track. Describing each car's behaviour relative to what's normal at that corner fixed it.
A fly brain that wouldn't steer: run as-is, the connectome saturated, with about a third of its neurons firing no matter where the target was. Correcting the membrane time constant to a realistic value gave sparse, meaningful activity and real left/right steering.
Real-time integration: telemetry, ML, dispatch, drone control, in-game cameras, the VLM and the dashboard all had to stay in sync without blocking each other.

Accomplishments that we're proud of

A complete end-to-end system: telemetry comes in and race control gets a drone on scene with a live feed and a report
Held-out Imola race (never seen in training): detected 45 of 64 incidents (70%) versus 39 for a rule-based baseline, gave 9 early warnings a median of 1.7 seconds ahead, and 11 of 13 predictions came true
Spa with new cars (GT3 and F1): twice the detections of the rules baseline, including every spin and off
A real fruit-fly brain flying a drone, where every neuron that lights up on screen is a real spike from the simulation

What we learned

Good features beat complex models. Designing around "normal for this corner" is what made the model work on new tracks.
Real-time AI is more than accuracy. Latency, data reliability, control and a clear interface matter just as much.
Biological simulations are fragile. Getting the fly brain working meant understanding the neuroscience behind it, not just treating it as a black box.

What's next for Fly Marshal

Earlier, more reliable predictions, so more incidents get a heads-up before they happen
Fewer false alarms and stronger contact detection
Wider VLM checks for oil, gravel on track and blocked racing lines
Real-world testing: take Fly Marshal off the simulator and onto real drones at real club circuits