-
-
Robo Mapper
Inspiration
Growing up in an agrarian family, we witnessed firsthand the importance of the harvesting process, which requires robust and accurate planning with immense labor and tedious work involved in agriculture. One of the most challenging and time-consuming tasks is collecting fallen nuts and fruits on the ground, a process often hindered by labor shortages. This personal experience became the core inspiration for my project. I realized there was a significant gap in accessible, offline-based intelligent simulators and on-demand tools for farmers, students, and researchers to experiment with autonomous solutions for this specific problem. I wanted to build a system that could address this real-world challenge by making robotics and AI more approachable.
What it does
Our project, the Offline Robo-Mind Mapper, is a comprehensive simulation framework for autonomous object collection. It allows users to: -> Generate Custom Maps: Users can type a natural language prompt to generate a custom 2D farm map, providing a tailored environment for their specific needs.
-> Integrate Real-World Data: Users can upload images of fallen objects, which the integrated YOLO model detects and counts. This data is then used as a heuristic for intelligent path planning.
-> Simulate & Benchmark: The simulator visualizes a planned path and simulates the robot's movement. This allows users to test and benchmark different path planning algorithms and assess their efficiency.
-> Prepare for Real-World Deployment: The simulator is specifically designed to be a testing ground that can be connected to a real robot via PySerial, allowing for seamless transition from simulation to physical deployment.
How we built it
The entire simulator is built with PyQt, which allows it to be packaged as a standalone offline .exe application. This was a critical design choice to ensure accessibility for users without an internet connection (best suited for farm environments). The core components of the system include:
GPT-OSS for Intelligent Map Generation: For the map generation, we used the GPT-OSS 20B model rendered offline via Ollama. The LLM takes a user's natural language prompt and generates a structured JSON object. This JSON data is then parsed by the application and used to plot a detailed 2D farm map with specific features.
YOLO for Perception and Heuristics: To provide the simulator with realistic perception, we manually annotated 9,000 images of fallen areca nuts and trained a custom model using YOLOv8. This trained model is then fetched through Roboflow, allowing the simulator to accurately detect and count nuts in images provided by the user. This nut count serves as a vital heuristic for the path planning algorithm.
Path Planning & Simulation: Once the data is in place, the user can select from different path planning algorithms. The simulator then calculates and visualizes the optimal path for the robot to collect the objects. This provides a clear way to compare algorithms and assess the efficiency of the mission in terms of time and energy.
Challenges we ran into
The biggest challenge was the integration of a physical robot. Due to unforeseen port connection issues and time constraints, we had to pivot and focus solely on the simulation aspect. While this was disappointing, it became a learning opportunity. It forced us to refine the simulator's features, making it a powerful and standalone tool for research and education.
Accomplishments that we're proud of
Offline, End-to-End System: We successfully built a complete AI system that runs entirely offline. From the LLM-powered map generation to the YOLO-based perception and the final simulation, every component is self-contained. This is a significant accomplishment that makes the tool highly accessible and reliable.
Complex Model Integration: We managed to seamlessly integrate three powerful and distinct technologies, like LLM for high-level reasoning, a custom-trained computer vision model for perception, and a path-planning algorithm, all into a single, cohesive application.
Bridging Data and Reasoning: We are particularly proud of our approach to using a reasoning-based LLM to generate the foundational map and then enriching it with real-world, data-driven perception from the YOLO model. This unique combination is the core innovation of our project.
Practicality and Real-World Applicability: The simulator is not just a proof-of-concept; it's a practical tool designed to solve a real-world problem. By using real annotated data and offering a clear, interactive interface, we've created a valuable resource for agricultural research and roboticists.
What we learned
Throughout this project, we learned the power and potential of combining multiple AI models into a cohesive framework. We discovered that large language models like GPT-oss aren't just for text generation; they can be powerful reasoning engines capable of creating structured data and even acting as a foundational layer for a robotics system in generating structured 2D maps. The necessity of an offline solution also taught us a great deal about local deployment and the importance of a robust, self-contained application. Most importantly, we learned that a well-designed simulator is a crucial research tool, providing a safe, efficient, and cost-effective environment for testing complex ideas before deploying them on a physical robot.
What's next for Offline Robo Mind-Mapper
Our immediate next step is to integrate a robotic arm to physically perform the object collection, transforming the simulation into a fully functional system. However, the true long-term vision for this project is to create an even more intelligent and adaptive robotic agent.
As a future enhancement, we plan to incorporate dynamic, real-time decision generation using the LLM. If the robot's onboard sensors and camera detect a new, unexpected obstacle or a new cluster of objects, this information will be fed to the gpt-oss model. The LLM would then act as the robot's brain, analyzing the new situation and generating a revised path or a new strategic plan on the fly. This would enable the robot to navigate dynamic, unpredictable environments, a crucial capability for real-world applications.
By having the LLM dynamically adjust the robot's behavior, we can move from a pre-planned system to a truly autonomous and reasoning-based agent, further pushing the boundaries of what is possible with offline AI in robotics.
Log in or sign up for Devpost to join the conversation.