PoleSentinel

Inspiration

Manual grid audits are slow, dangerous, and expensive. Climbing poles or sending bucket trucks to survey equipment takes time, especially when assessing infrastructure after severe weather. We wanted to automate this process for Dominion Energy to make inventorying assets, like transformers and crossarms, and estimating their heights safer and faster.

What it does

PoleSentinel is an automated grid-auditing prototype that pairs a custom YOLO computer vision model with an Arduino Nano 33 to instantly inventory utility assets. From a single camera feed, it successfully identifies essential elements like poles, crossarms, and transformers, and automatically calculates total pole height (with ± uncertainty on every number) and locations, measurements, and heights of every transformer and crossarm.

How we built it

Initially, we tried using the Gemini API for image recognition, but we quickly pivoted to training a custom YOLO model for faster, localized bounding-box detection. We started by training on a massive 2,400-image Roboflow catalog. While it accurately identified large elements like poles and crossarms, it failed on smaller or less frequent objects (like wires and transformers) due to inconsistent community labeling.

Realizing the data was the bottleneck, we restarted from scratch. We collected ~50 original photos of utility poles around Columbia, SC, and merged them with 260 images from a secondary wire-focused dataset. We manually annotated the entire batch from scratch, hand labeling Poles, Wires, Crossarms, and Transformers.

To estimate the height of the poles and attached elements, we extract the polgygon masking coordinates from our YOLO predictions. By referencing a standard physical utility pole height (P), we project the physical height of any attachment (E) using vertical ratio mapping:
$$ E = P \times \frac{B-A}{B-T} $$ E = Estimated physical height of the attachment P = Physical reference height of the standard utility pole B = Y-coordinate of the base of the pole in the image A = Y-coordinate of the attachment in the image T = Y-coordinate of the top of the pole in the image

Notably, this method requires us to already know the pole's reference height (P), which caused massive measurement errors. A photo alone cannot tell you true scale. To solve this and measure the actual physical height of the pole, we built a handheld hardware rig to capture physical telemetry. When a photo is taken, a 24 GHz radar measures the exact distance to the pole (D), and an IMU sensor records the camera's physical tilt. By combining the AI bounding boxes with the exact camera tilt, we get the true optical angles to the top and base of the target. Using standard trigonometry, we calculate the exact physical height (H) in meters:

$$ H = D \times (\tan(\theta_T) - \tan(\theta_B)) $$

H = Total physical height of the target in meters D = Exact distance to the pole measured by the 24 GHz radar $\theta_T$ = True angle to the top of the target $\theta_B$ = True angle to the base of the target

Challenges we ran into

One task that was particularly challenging was building an AI model that could accurately isolate the different parts of electric poles. Our first two models struggled with precision, often leaving elements unmarked while hallucinating unrelated objects. Fixing this required two of our members to spend the entire evening hand-annotating ~300 images, followed by another 2 hours of running training epochs.

We also ran into strict hardware limitations. Running a computer vision pipeline and estimating 3D spatial height from a 2D image is incredibly taxing. We had to creatively work around the memory and processing constraints of the Nano 33, relying on our polygon masking math because a true dual-camera depth setup was too heavy for our hardware to process efficiently.

Accomplishments that we're proud of

We're incredibly proud that we were not only able to build our own training set within 12 hours, but that we successfully integrated a functional computer vision pipeline with our physical hardware. Walking around Columbia to manually build a dataset taught us that throwing more data at a model doesn't work if the data is low-quality. We are extremely proud of the final accuracy of our custom YOLO model.

What we learned

Going into this project, we were all unfamiliar with training our own models, much less combining one with an Arduino microcontroller to take hardware inputs. In addition to learning how to operate Roboflow, we gained deep, hands-on experience in curating, cleaning, and annotating our own data to solve real-world AI problems.

What's next for PoleSentinel

To elevate our prototype to an enterprise-grade auditing tool, we will migrate our edge compute to a Raspberry Pi 5 running a QNX Real-Time Operating System (RTOS) to guarantee the deterministic hardware-software synchronization required for our exact trigonometric height calculations $H = D \times (\tan(\theta_T) - \tan(\theta_B))$. Furthermore, we will upgrade our handheld sensor payload by integrating a GPS pinger for precise geospatial asset mapping, replacing the wide-beam radar with a LiDAR Time-of-Flight rangefinder for exact crosshair targeting, and swapping our IMU for a drift-free 9-DOF BNO085 with on-board sensor fusion