Agric_orion

Architecture of Agric_Orion

Inspiration

Agric-ORION started with a question: what if an autonomous robot could understand what a person wants it to do, make sense of its surroundings, and adapt to finish the job safely?

Most autonomous systems are built around fixed goals, fixed paths and expected conditions. The real world doesn't cooperate. In farming, something can block a route, field conditions change, and a robot meets situations nobody planned for. We wanted to explore how to move past rigid, pre-programmed behaviour, towards robots that can reason about their mission and respond when conditions change.

So we built Agric-ORION, a research-driven project that brings together agentic AI, computer vision and mobile robotics. A person gives a high-level mission, such as inspecting a section of a field. The system works out what that means, gathers information about the physical world around it, and hands the actual driving to proven, deterministic navigation.

Farming is where we started because it is a hard place for robots: obstacles, uneven ground and a lot of uncertainty. We see it as a way to explore technology that could one day support field inspection, environmental monitoring and other farm work.

One principle sits at the centre of the project: intelligence and physical control need clearly separate responsibilities. The AI can interpret a mission and reason about what the robot observes. Established navigation and control systems stay in charge of movement. By combining OpenCV-based vision with the robot's laser and other sensors, we look at how different kinds of evidence can improve what a robot understands about its surroundings, and change what it decides to do.

Agric-ORION isn't only about making a robot move on its own. It's about autonomous systems that understand context, adapt, and turn human intentions into safe, useful action. Agriculture is the starting point, but the same idea applies to industrial inspection, environmental monitoring, logistics and anywhere else autonomous vehicles have to turn what people want into safe action.

What it does

You tell the robot what you want in plain words, like "Go to the field centre." An AI assistant works out what that means, and the robot's normal navigation system does the driving.

The new part is eyes. A camera watches the path ahead, and OpenCV 5, running on AWS, turns each picture into a simple answer: is something in the way, and how far away? If there is, the robot stops and waits. When the way is clear again, it carries on by itself.

One rule matters most: the AI can suggest, but it can never overrule a stop. In our tests the AI did suggest carrying on while the camera could see the box. The rule stopped it every time, in all five runs we did with the real AI assistant.

A surprise that shaped the project

While testing, we noticed something odd. The robot's navigation reported "mission complete" while the robot was pushed up against a box, more than a metre from its goal. Its wheels were still turning, so it believed it was still moving.

That changed what we cared about. Making the robot move was only half of it. The other half was making it honest about whether it could.

How we built it

We built a simulated four-wheel farm robot in Gazebo, driven by ROS 2 and Nav2. Its laser scans one flat slice about 20 cm above the ground, so low objects slip underneath it.

The vision part is deliberately simple. A short OpenCV program looks at the ground ahead, picks out objects that don't belong, and estimates how far away they are. It runs as a small function on AWS Lambda. We added a second check for gray objects, which the first one couldn't see.

The AI assistant is Amazon Nova 2 Lite on Amazon Bedrock. Its answers are treated as untrusted: a guard written in code and a separate checker look at everything before the navigation system gets to act on it.

What we found

All of this is in simulation, with small numbers of runs, so it shows how the idea works rather than how often.

The laser missed every test box up to 18 cm tall. The camera saw all of them, from 3 to 50 cm.
In every run where the matching camera check was on, the robot stopped and then carried on by itself once the box was removed (10 of 10). With it off, the robot drove into the box (4 of 4) and still reported success.
Twice we read the robot's true position and compared it with what it believed. It was 1.44 m and 1.26 m from its goal while reporting that it had arrived.

Challenges we ran into

Our first camera check missed a gray box, and we chose to document that instead of tuning it away. The second check catches gray boxes, but two of our own predictions about it turned out wrong, so we still don't know exactly where it fails.

Sending pictures to the cloud adds delay, about a second in our tests, so the robot notices things a little later than it would with the processing on board.

Our first choice of AI model was blocked by a billing restriction, so we built the assistant behind a swappable interface and used a labelled stand-in until we could switch to Bedrock.

We also threw out some of our own measurements when we found we had taken them at the wrong moment.

What we learned

Safety rules belong in code, not in an AI's instructions. A "success" message is not proof of success. And it is worth labelling what is real and what is a stand-in, and reporting the failures next to the results.

What's next

Real field images instead of a simulation, a vision check that copes with shadows and texture, an AI assistant that can call the vision tool itself, and other places where robots have to turn what people want into safe action.