Frontier RoboFarm

RoboFarm brings Physical AI to agriculture, using NVIDIA Isaac Sim, Nemotron, Cosmos, and Nebius to power autonomous crop inspection and deliver actionable insights to farmers.

Comment

Inspiration

Agriculture is the backbone of many economies around the world, supporting millions of livelihoods and playing a vital role in food security. Yet many essential farming tasks still depend on repetitive manual labor. In tomato greenhouses, farmers must inspect hundreds or thousands of plants individually, monitoring their health, checking fruit ripeness, and identifying problems before they spread.

We started Frontier RoboFarm with a simple question: How can we help farmers monitor thousands of plants, detect problems early, and make better decisions without time-consuming manual inspections? The solution must also be affordable, practical, and accessible to farmers everywhere.

Starting with real greenhouse environments in Thailand, we set out to build an autonomous robot that could navigate independently, inspect crops, collect consistent plant-level data, and transform observations into actionable insights for farmers. While Thailand is our starting point, the challenges we aim to address are shared by farmers across the world.

Our ambition goes beyond building a robot. We designed and assembled RoboFarm ourselves using affordable, commercially available components, with local sourcing wherever practical. We want farmers, researchers, and developers to be able to build, repair, and adapt agricultural robots without depending entirely on expensive proprietary machines. Our long-term plan is to share RoboFarm as an open-source hardware project.

RoboFarm represents our vision of technological self-reliance — bringing Physical AI out of research labs and into real farms, making advanced robotics accessible to the agricultural communities that sustain economies and feed the world.

What It Does

Frontier RoboFarm moves through greenhouses on its own to inspect plants up close, assess tomato ripeness and visible damage, and report which areas need a farmer’s attention. It brings cameras close to the crops, capturing detailed views of leaves and fruit from multiple heights and angles.

Travels to the Plants Autonomously

The robot navigates greenhouse aisles and travels between greenhouses to carry out a survey. It stops at inspection locations, can return to an area for another look, and returns to its dock after completing its mission.

Examines Leaves and Fruit Up Close

It captures plants from different heights and angles on both sides of the aisle, collecting close-range images of leaves, stems, and fruit. These images support screening for visible problems such as yellowing leaves, leaf spots, and fruit lesions.

The photographs are recorded with their locations, giving farmers evidence they can review and a place they can return to.

Assesses Harvest Readiness and Visible Crop Condition

RoboFarm estimates how many observed tomatoes are ripe, turning, or green, and flags fruit with visible lesions or damage. This helps farmers understand what appears ready for harvest and which areas require further inspection.

Harvest readiness and plant health are assessed separately. Counts describe what the cameras observe; they are not a guaranteed inventory of fruit hidden behind foliage.

Builds a 3D Understanding of the Greenhouse

As it travels, RoboFarm collects spatial observations and reconstructs a 3D map of its surroundings. Mapping helps establish where the robot has travelled and how greenhouse spaces connect.

Repeated surveys provide additional observations for comparing the environment over time. Continuously improving a persistent map across separate missions is a further development goal, rather than a capability we have already fully validated.

Responds to Situations Along Its Route

The robot checks the path ahead for obstacles and uses visual observations to interpret situations such as a person or equipment blocking its route. Its control and decision-making system can stop movement, wait, or request human assistance instead of continuing blindly.

Returns to Investigate Uncertain Findings

When an observation needs confirmation, RoboFarm can revisit the recorded location and capture more images. This allows the inspection to include follow-up, rather than ending with the first photograph or model prediction.

Reports What Matters to the Farmer

After a survey, RoboFarm produces a report in English or Thai containing:

  • Estimated ripe, turning, and green tomato counts.
  • Observed fruit lesions or damage.
  • Potential plant-health problems.
  • Photographs and locations supporting the findings.
  • Revisit results and unresolved observations.
  • Suggested checks and situations requiring human attention.

The intended benefit is to reduce repetitive scouting and help farmers focus their time on harvesting, checking suspicious plants, and addressing problems. Autonomous movement brings the robot close to the crops; recorded images provide evidence; mapping gives that evidence a location; and decision-making enables follow-up.

How We Built It

Robot Hardware and Sensing

We designed and assembled RoboFarm ourselves around a custom four-wheel skid-steer mobile platform, approximately 0.53 m wide, with a navigation footprint of approximately 0.75 m. We selected modular, commercially available parts to keep the robot affordable and make assembly, replacement, and local maintenance practical.

The robot uses independently controlled left and right wheels for differential steering. Because skid-steer motion introduces wheel slip, particularly during turns, the navigation system combines multiple sensor measurements to estimate its position.

Our sensing architecture includes a 360-degree 2D LiDAR, wheel odometry, an inertial sensor, six plant-inspection cameras, and a forward-facing camera. The R181-S configuration additionally incorporates a ZED 2i stereo camera for visual localisation and depth estimation.

The six inspection cameras are mounted at three heights on both sides of the robot, approximately 0.79 m, 1.21 m, and 1.64 m above the ground, capturing portrait images at 1080 × 1920 resolution.

We selected NVIDIA Jetson Orin as the target onboard computing platform, while computationally intensive simulation, perception experiments, and AI inference are supported by Nebius cloud infrastructure.

Digital Twin and Physics Simulation

We developed a full-farm digital twin using Blender and NVIDIA Isaac Sim, based on a real agricultural site in Thailand. After contacting the farm and obtaining site information, we collected ground-level photographs, aerial imagery, and layout references to reconstruct the agricultural environment. Our objective was to create a realistic virtual testing ground where we could develop, evaluate, and refine RoboFarm before deploying the physical robot at the actual site.

The digital twin covers the broader farm environment, including multiple greenhouse clusters, cultivation areas, agricultural ponds, farm buildings, internal roads, and connecting pathways. We reconstructed the surrounding agricultural infrastructure to represent the conditions the robot may encounter while operating across the farm.

Inside the greenhouses, we developed a detailed tomato cultivation environment for autonomous navigation and plant-inspection experiments. This environment contains 672 plant instances, including nine synthetic anomaly plants representing chlorosis, leaf spots, fruit lesions, wilting, and defoliation. These controlled variations support repeatable experiments with different plant conditions.

We use NVIDIA Isaac Sim to simulate RoboFarm's physical behavior and sensor observations. NVIDIA PhysX handles robot dynamics at a 120 Hz physics simulation rate, while simulated LiDAR and RTX-rendered cameras provide sensor data for navigation and computer vision experiments. This allows us to evaluate robot movement, sensor configurations, and plant-inspection systems within the reconstructed farm environment.

Navigation and Localisation

RoboFarm combines NVIDIA cuVSLAM, 2D LiDAR, wheel odometry, and gyroscope measurements for autonomous navigation and localisation. NVIDIA cuVSLAM processes stereo images from a ZED 2i camera with a 12 cm baseline, providing metric-scale visual motion estimates. Running on NVIDIA Jetson Orin Nano, cuVSLAM achieved 188 FPS in stereo mode and 82 FPS with stereo-inertial input, supporting the robot's 10 Hz navigation control loop.

LiDAR scan matching corrects accumulated positioning drift against the farm map, while wheel odometry and gyroscope measurements provide motion and heading information. Inside greenhouse aisles, LiDAR also measures the robot's lateral position and alignment relative to planting-bed boundaries. Combining these measurements with cuVSLAM helps address wheel slip and localisation uncertainty in repetitive greenhouse environments. In our simulation experiments, sensor fusion reduced in-aisle localisation error by approximately 12% over a 111 m route.

Stereo camera observations also support 3D mapping, allowing RoboFarm to reconstruct and refine its spatial representation of the farm during surveys.

The navigation system operates using simulated sensor observations, with NVIDIA Isaac Sim's ground-truth position recorded separately to evaluate localisation accuracy.

A mission-control system coordinates inspection waypoints, greenhouse transitions, obstacle handling, and return-to-dock operations. HERMES provides high-level mission requests through a validated interface, while onboard navigation and motion control manage the robot's movement and safety.

HERMES serves as RoboFarm's mission-planning and decision-making agent, using the farm map and inspection results to coordinate surveys, revisit flagged plants, and determine when to return to the dock or request human assistance. Its decisions are executed through a validated command interface, while the robot's onboard navigation system handles movement and safety.

For cost-sensitive deployments, RoboFarm also supports autonomous navigation using LiDAR, wheel odometry, and gyroscope measurements without a stereo camera. This configuration retains the core navigation capabilities, while stereo vision and NVIDIA cuVSLAM provide additional localisation accuracy and visual mapping capabilities when available.

3D Mapping and Reconstruction

RoboFarm uses lingbot-map for 3D reconstruction of the farm environment, using camera observations to build detailed spatial representations that support map refinement and future navigation.

We also evaluated MASt3R-SLAM, MapAnything, VGGT, and SLAM3R. These approaches struggled in our greenhouse experiments due to repetitive plant structures and visually similar rows. As the robot moved deeper into the greenhouse, the models increasingly confused similar-looking plants and locations, leading to inconsistent spatial reconstruction and localisation drift.

lingbot-map produced the most usable reconstruction results in our experiments, making it our selected approach for building and refining RoboFarm's 3D farm maps.

Detection and Multi-View Reasoning

We fine-tuned RT-DETRv2 using 1,942 training images combining synthetic renders from NVIDIA Isaac Sim with public agricultural photographs. The detector identifies tomato ripeness, fruit lesions, and visible plant abnormalities. Synthetic data generation provides annotations directly from the simulation, reducing manual labelling effort.

For plant-level analysis, RoboFarm captures six camera views at each inspection location and processes them together using NVIDIA Nemotron 3.5 Super VL through Nebius Token Factory. The model combines observations from multiple angles to assess plant conditions, identify potential problems, and generate structured findings with recommended follow-up actions. Spatial information from the inspection locations also helps reduce duplicate fruit counting across overlapping images.

NVIDIA Cosmos Reason 2 processes forward-camera observations to interpret the surrounding environment, including people, equipment, doorways, and unexpected obstacles. These observations support HERMES in coordinating appropriate responses during robot operations.

Agent and Cloud Pipeline

We integrated HERMES Agent with RoboFarm to coordinate inspection missions and respond to findings. HERMES uses NVIDIA Nemotron through Nebius Token Factory and our custom skills to request surveys, revisit plants requiring further inspection, return to the docking station, and notify farmers. Each request is validated by the robot's mission-control system before execution.

Our cloud infrastructure on Nebius connects robot operations with GPU-accelerated data processing. Each survey records sensor images, location information, and mission events, which are processed through object detection, multi-view reasoning, 3D mapping, and environmental analysis.

The pipeline tracks each processing stage independently, allowing completed results to be reused and supporting recovery from individual processing failures. The final output is a structured report in Thai or English, containing crop observations, estimated harvest readiness, plant-health findings, and recommended actions.

Challenges We Ran Into

Initial Localisation and Positioning Accuracy

Determining the robot's initial position was a major challenge. We addressed this by starting each survey from a known docking location and continuously correcting its position using LiDAR scan matching against the farm map. Our system evaluates scan quality before accepting corrections, achieving approximately 98% beam agreement within 7 cm during typical matching steps. In simulation, the robot achieved an overall localisation error of approximately 0.26 m.

Navigation Through Repetitive Greenhouse Environments

Greenhouse aisles contain visually similar plants and structures, making localisation difficult. We combined wheel odometry, gyroscope measurements, LiDAR, and NVIDIA cuVSLAM to maintain positioning accuracy. cuVSLAM achieved 188 FPS on NVIDIA Jetson Orin Nano, while LiDAR-based map corrections helped resolve ambiguity between similar greenhouse structures. Several camera-only mapping approaches struggled with accumulated drift, reinforcing the importance of sensor fusion.

Building a Reliable Digital Twin

Creating a realistic simulation from photographs of our actual farm required accurate geometry, materials, lighting, and sensor configurations. Integrating Blender with NVIDIA Isaac Sim introduced challenges involving scale consistency, USD scene organisation, and physics compatibility. We also used Nebius GPU infrastructure to render 20 camera views across recorded robot trajectories, enabling detailed perception evaluation without maintaining our own rendering cluster.

Decision-Making with Limited Onboard Compute

Running large AI models on a compact robot is challenging due to limited onboard computing resources. We addressed this by separating real-time navigation from high-level decision-making. RoboFarm handles motion control locally, while HERMES uses NVIDIA Nemotron through Nebius Token Factory to determine mission-level actions, such as revisiting plants or notifying farmers. During our recorded simulation, HERMES requested two follow-up inspections and cleared two false alarms. If cloud connectivity is lost, the robot can continue its predefined mission, maintain local safety controls, store collected data, and return to its dock.

Training on Simulation Alone Was Not Enough

Our first detector was trained only on Isaac Sim renders, labelled automatically — nobody drew a box. It looked good in the twin (mAP50 0.45) and failed on real photographs: 0.39 on real tomato fruit and 0.005 on real leaf disease — effectively blind to yellowing and spots on real leaves. The fix was not more simulation but mixing in real images: retraining on the same renders plus the public AgRob and PlantDoc photo sets doubled real-fruit accuracy to 0.75 and lifted real leaf-disease detection to 0.50, with no loss in the twin (0.48). The lesson shaped the whole project: simulation gives volume, labels and control; a small amount of real data gives the model its eyes. That is the detector the robot runs today, and the reason we treat field photos from our own farm as the next most valuable thing to collect.

Accomplishments That We're Proud Of

1. Fully Autonomous Greenhouse Survey

RoboFarm completed a 117 m survey across two tomato greenhouses in 12.5 minutes, performing 161 inspection stops and returning to its docking station without human intervention or collisions. The robot navigated entirely using simulated sensor observations, achieving 0.26 m localisation RMSE against recorded ground truth.

2. End-to-End AI Inspection and Reporting

Our system connects autonomous navigation, six-camera plant inspection, AI reasoning, and farmer reporting into a single workflow. Across 966 images, the combined RT-DETRv2 and NVIDIA Nemotron pipeline identified 6 of 9 synthetic anomaly plants. HERMES also requested two follow-up inspections to resolve uncertain findings before reporting. Each finding retains its location, supporting images, and recommended actions.

3. Automatically Labelled Synthetic Training Data

Our digital twin serves as both a robotics testing environment and a source of training data. Using automatic annotations generated from the simulation, we produced 1,942 labelled training images without manual bounding-box annotation. Combined with public real-world agricultural photographs, this enabled us to fine-tune RT-DETRv2, achieving mAP50 scores of 0.75 for fruit detection and 0.50 for leaf disease detection on real images.

4. Reproducible Deployment on NVIDIA Hardware and Nebius

We deployed our robotics software on a real NVIDIA Jetson Orin Nano, achieving 188 FPS with stereo cuVSLAM. Our cloud pipeline runs on Nebius RTX PRO 6000 GPU infrastructure and completed the survey-to-report workflow in 9 minutes and 59 seconds, producing reports in Thai and English. The project also passed 114 automated software tests.

Navigation and localisation accuracy were evaluated using simulated sensor data. Stereo cuVSLAM error was measured after Sim(3) alignment. Plant-anomaly results come from a controlled simulation experiment and require further field validation.

What's Next for Frontier RoboFarm

Our next milestone is to bring RoboFarm from simulation into real-world agricultural environments, validate its performance, and expand its capabilities toward practical farm operations.

1. Real-World Testing and Validation

We plan to deploy RoboFarm at the agricultural site used to develop our digital twin. Using NVIDIA Jetson Orin Nano and ROS 2, we will evaluate autonomous navigation, sensor accuracy, obstacle handling, and operational reliability under real greenhouse conditions.

2. Improving Navigation Across Different Environments

We aim to improve RoboFarm's ability to operate across different farm layouts and challenging conditions, including uneven terrain, wet surfaces, changing lighting, narrow pathways, and unexpected obstacles.

3. Expanding to More Crops

Starting with tomatoes, we plan to extend RoboFarm's inspection capabilities to other crops, including lettuce and nursery plants. This will involve adapting camera configurations, detection models, and agricultural knowledge to different growing environments.

4. Robotic Manipulation and Active Inspection

We plan to integrate a robotic arm, enabling RoboFarm to interact with plants, lift leaves, inspect hidden areas, and position sensors closer to specific plant structures. This will extend its capabilities from visual monitoring to active plant inspection.

5. Improving Reliability and Deployment Readiness

We will continue refining the AI agent, robotic hardware, and cloud infrastructure to support reliable long-term operation, recovery from failures, and human oversight. Our goal is to make RoboFarm easier to deploy, maintain, and operate across multiple farms.

We also plan to release RoboFarm’s designs, component list, assembly instructions, and software as an open-source project, so others can reproduce and adapt the platform to their own farms. Our long-term vision is to make Physical AI a practical and affordable tool for agriculture, with robots that can be assembled, maintained, and improved using locally available resources.

Built With

Share this project:

Updates

Submission history