Inspiration
Manual road infrastructure surveys are slow, labor-intensive, and highly reactive. The goal was to transform standard vehicles into automated, real-time edge-compute monitors that bridge the gap between embedded physical electronics and high-speed cloud AI, allowing cities to detect and dispatch repair crews before hazards cause damage.
What it does
Pothole Sentinel is an automated, edge-to-cloud road hazard monitoring system. When its embedded Arduino detects a physical impact, it instantly triggers a webcam to capture the road surface. It then uses Nebius vision AI to assess the pothole's severity and NVIDIA Nemotron AI to automatically generate municipal repair instructions in under 3 seconds, closing the loop by lighting an LED on the hardware device to alert the driver.
How it was built
Pothole Sentinel operates as a complete edge-to-cloud-to-edge pipeline:
Hardware Edge Trigger: An Arduino Nano ESP32 acts as the physical edge node. To simulate a mechanical road impact, a timer loop triggers wire pins to fire a physical serial interrupt (EVENT:IMU_BUMP) into the local machine. To guarantee ultra-low latency, we innovatively implemented an Interrupt RAM Service approach. By forcing the microcontroller to load the interrupt service routine directly into its Instruction RAM (IRAM), we completely bypassed standard flash memory read delays. This ensures the edge device registers the physical "impact" instantaneously, shaving off critical microseconds at the very beginning of the pipeline.
Vision & Cloud Inference: A Python edge script (sentinel.py) catches the serial interrupt and instantly wakes a webcam to capture the road surface. The image is passed to the Nebius Token Factory running the MiniCPM-V-4_5 vision model, which grades the hazard severity.
Automated Dispatch: The structured severity payload is piped into NVIDIA Nemotron-3.5-Lightning, acting as a virtual dispatcher to generate immediate municipal repair instructions.
Physical Actuation: The edge script parses the severity and sends a command back over the serial port to the Arduino, actuating an RGB LED to alert the driver. All operational metrics are logged locally to incident_telemetry.json.
Challenges I ran into
The primary hurdle was establishing reliable, low-latency, two-way communication between the embedded microcontroller and the cloud APIs. Ensuring the Python script could listen for serial interrupts without blocking the webcam capture or dropping the API connection required careful thread management. Additionally, forcing the vision model to reliably return a strictly formatted JSON payload for the NVIDIA model to parse required precise prompt engineering.
Accomplishments that we're proud of
We are most proud of engineering a true Hardware-in-the-Loop pipeline that reliably chains physical edge hardware with two distinct cloud AIs in under 3 seconds. Specifically, we are proud of successfully implementing the Interrupt RAM Service (IRAM) on the Arduino to achieve near-instantaneous hardware triggers, and seamlessly routing structured JSON from the Nebius vision model directly into the NVIDIA dispatch model without latency bottlenecks.
What we learned
This project demonstrated how to successfully design a Hardware-in-the-Loop (HIL) system, specifically orchestrating local physical hardware logic with high-throughput cloud models. It also required optimizing API calls to ensure the total system latency remained viable for a moving vehicle. The critical time constraint required keeping the end-to-end loop below 3 seconds, modeled as:
t_system = t_serial + t_vision_api + t_nemotron + t_actuation <= 3 seconds
What's next for Pothole-Sentinel
Hardware IMU: Replace the simulated timer with a physical 6-axis IMU to trigger the IRAM interrupt purely on genuine G-force impacts.
GPS Telemetry: Integrate a NEO-6M GPS module to append precise latitude and longitude coordinates directly to the AI dispatch payload.
Edge Pre-filtering: Deploy a lightweight local model (like YOLOv8-nano) to filter out empty frames, sending only confirmed hazards to the cloud APIs to slash latency and costs.
Municipal Dashboard: Build a live web map to aggregate, map, and cluster the AI-generated dispatch tickets into prioritized repair zones for city crews.
Log in or sign up for Devpost to join the conversation.