Inspiration
In manufacturing environments, workers often perform repetitive material-handling tasks, such as picking up objects and transporting them between workstations.
Autonomous robots could help with these tasks, but reliable manipulation depends on accurate visual perception. Even small errors in object detection or position estimation can lead to unsuccessful grasps.
This inspired me to develop Vision2Action, a system that combines autonomous robotic manipulation with measurable, reproducible computer vision evaluation.
Rather than focusing only on whether a robot can complete its task, I wanted to understand how accurately its perception system performs.
What it does
Vision2Action is an OpenCV 5-powered robotic perception system integrated with a simulated Galbot mobile manipulator.
It enables the robot to:
- Navigate autonomously between pickup and drop-off locations.
- Detect and segment objects using OpenCV 5 and YOLO11.
- Estimate object positions and orientations using camera data.
- Plan and execute robotic grasping and placement.
- Evaluate perception performance using simulation ground truth.
- Reproduce perception evaluations using AWS.
The robot runs locally in MuJoCo simulation, while AWS provides a separate cloud-based evaluation pipeline.
How we built it
OpenCV 5 — Robotic Perception
I exported my YOLO11 segmentation model to ONNX and implemented inference using OpenCV 5's DNN module. The pipeline handles image preprocessing, neural network inference, object detection and segmentation mask reconstruction. The resulting segmentation masks are used to estimate the object's position and orientation. These estimates are then passed to the robot's manipulation system.
ROS 2 — Autonomous Manipulation
The system uses ROS 2 Jazzy to integrate perception, navigation and manipulation. Nav2 handles autonomous navigation, while MoveIt 2 plans the robot's arm movements. The robot is simulated in MuJoCo, allowing the system to be tested against known object positions.
AWS — Cloud Evaluation
I developed an AWS-based evaluation pipeline to make perception testing more reproducible. Evaluation data is uploaded to Amazon S3, which triggers an AWS Lambda function running OpenCV 5. The cloud application processes the evaluation images and generates reports containing perception accuracy metrics. The cloud evaluation runs independently of the complete robot simulation.
Results
OpenCV 5 Inference
On a dataset of 150 validation images:
- OpenCV 5 produced segmentation masks identical to the original Ultralytics implementation.
- Median local inference time decreased from 25.3 ms to 20.9 ms.
- This represents a 17.4% reduction in median inference latency under the tested conditions.
Autonomous Robot Demonstration
In a successful simulated mission:
- 154 camera frames were processed using OpenCV 5.
- Median live frame processing time was 42 ms.
- Object position estimation error was 1.71 mm.
- Object orientation estimation error was 1.02°.
- Final placement error was 4.2 mm.
AWS Evaluation
For the exported evaluation images:
- Segmentation IoU: 0.896.
- Orientation error: 1.27°.
- AWS and local evaluation produced matching accuracy results.
These results demonstrate that the OpenCV 5 evaluation can be reproduced in a different computing environment.
Challenges we ran into
One major challenge was integrating OpenCV 5 into an existing ROS 2 system without affecting its dependencies. To address this, I developed an isolated OpenCV 5 inference service that communicates with the robot's perception pipeline.
Another challenge was ensuring that the ONNX model produced the same segmentation results as the original implementation. This required careful handling of preprocessing, inference outputs and segmentation masks.
For AWS deployment, I also had to package OpenCV 5 and its dependencies into a container suitable for AWS Lambda.
Accomplishments that we're proud of
Learning AWS cloud computing: This was my first experience working with AWS. I learned how to use Amazon S3, AWS Lambda and Amazon ECR to deploy a cloud-based evaluation pipeline and generate reproducible results.
Integrating OpenCV 5 with ROS 2: Successfully integrated OpenCV 5 into an existing ROS 2 robotic system while managing dependency compatibility issues.
Building a working robotic demonstration: Successfully demonstrated autonomous navigation and pick-and-place using OpenCV 5 for live object detection and segmentation.
What we learned
This project helped me better understand how computer vision models can be integrated into complete robotic systems.
I also gained experience deploying containerised applications on AWS and developing repeatable evaluation workflows.
More importantly, I learned that successfully completing a robotic task is only one part of building a reliable system. Measuring its performance and understanding its limitations are equally important.
What's next for Vision2Action: Cloud-Validated Robotic Manipulation
The current system has been evaluated in simulation, so its performance on physical hardware has not yet been established.
Future work includes:
- Testing with real robot cameras.
- Evaluating under different lighting conditions and object arrangements.
- Improving robustness to moving or partially occluded objects.
- Deploying the system on a physical robot.
- Expanding the cloud evaluation to support additional objects and datasets.
Log in or sign up for Devpost to join the conversation.