Evaluation is all you need

With heavy machinery, the safety of an autonomous driving system is as important as its accuracy. We use artificially generated data to determine conditions where the machine can run with confidence.

Evaluation pipeline

We have developed a system to determine the safety boundary of a perception system for autonomous driving. The system is build up like this:

At its core, we use the Farming Simulator to generate a variety of scenes. Next, we evaluate the model on this data to determine under which conditions it works well. This is useful in all three stages of the product lifecycle:

1. During development, it helps determine where the system can be improved.
2. During verification, it can be used to determine if the system works in the entire ODD (Operational Design Domain)
3. On the field, it can be used to disable the autonomous system once the safe conditions are not met anymore.

For the demo, we used the evalution pipeline on a simple YOLOv8 detector (yolo_detector.py). The code instrumentation for the Farming Simulator is in changecrop.py.

Built With

• farming-simulator
• python