Inspiration :Weeds are one of the biggest threats to crop yield, and the answer is still to spray entire fields with herbicide. That is costly, harms soil and water, and drives herbicide-resistant weeds. Manual weeding is precise but needs increasingly scarce labour. If a robot could look at every plant, decide whether it is a crop or a weed, and act only on the weeds? The missing piece is data. Labelled field images are expensive to collect, vary by crop, season, and lighting, and rarely cover hard cases, such as small weeds touching a crop. Physical AI in simulation, trained on GPUs, looked like the way to close that gap.
What it does: It is an autonomous weeding rover that :
1) Detects and segments crops and weeds at the individual-plant level from camera frames Converts detections into ground coordinates and targets only the weeds 2) Skips any action when confidence is low, or a weed is too close to a crop, logging it instead of risking damage 3) Produces a live field map and an auto-generated report of weeds removed, skipped cases and herbicide avoided
How we built it :
Simulation. We built a crop field in NVIDIA Isaac Sim with rows of crops and randomly placed weeds of several species and growth stages. Synthetic data: Using Replicator, we generated labelled images with randomised lighting, soil, shadows, plant size and occlusion, with perfect masks at no labelling cost. Training on Nebius: We trained the segmentation model on Nebius GPU instances, running N experiments, including ablations with and without domain randomisation. Optimisation: We exported the model with TensorRT and benchmarked it at ms per frame . Localisation and targeting.: Camera calibration and rover odometry convert pixel detections into field coordinates, and a planner times the tool so it lands on the weed as the rover moves. Confidence-gated control: A safety layer enforces a margin around crops and skips uncertain detections. Dashboard: A live view shows overlays, the weed map, and a comparison of targeted removal against blanket spraying. Challenges we ran into The sim-to-real gap: A model that scores well on synthetic scenes can fail on real
Challenges we ran into:
The sim-to-real gap. A model that scores well on synthetic scenes can fail on real images. We addressed it with heavy domain randomization and a small real-image validation set, and we report plainly Weeds that look like crops. Early growth stages are visually close. Adding hard negatives and training across growth stages improved Small objects at low resolution. We missed tiny weeds until we added [tiled inference or higher-resolution crops]. Timing. Detection, rover motion and tool action are offset in time, so we modelled the latency to hit moving targets accurately. Safety versus coverage. Removing more weeds meant risking more crop damage. The confidence gate let us tune this trade-off deliberately
Accomplishments that we're proud of
[10]%Weed removal with only [12]% crop damage in simulation [10]% less herbicide-treated area than blanket spraying
What we learned:
Synthetic data is only as good as its variety. Randomization choices mattered more than adding raw volume. For Physical AI, the action layer matters as much as the model. A good detector still fails without precise localization and timing. Building in a confidence-gated “do nothing” option made the system safer and easier to trust. GPU infrastructure shortened the loop: parallel training and quick ablations on Nebius let us iterate on data instead of waiting on jobs. Honest evaluation, including reporting what did not transfer, is more convincing than a single headline number.
What's next for AgriSight Robotics :
Validate on real field imagery and a physical rover Add growth-stage awareness and a weed-density prescription map for farmers Extend to multi-rover coordination across large fields
Log in or sign up for Devpost to join the conversation.