AI Powered Mars Rover Robot

Leveraging AI to detect and store extraterrestrial objects!

Comment

Inspiration

This project was majorly inspired by the designated theme of space. The NASA Mars rovers served as a further inspiration and we even have "Mars Rover" in our project name.

What it does

The rover can move in any direction using a web joystick. It has a wireless camera attached to it which enables it to detect any object (debris and rocks are some applicable objects for the space theme). The image by the wireless camera is then processed in a LLM (Llama 4.1-Maverick). If the LLM can't detect an object, a picker picks the object and puts it in the main body of the rover for storage. The moving of the rover, taking images from the camera and putting it into the LLM, and picking and storing objects are all manually controlled by us and not autonomously.

How we built it

We built this project by first creating the designs (both hardware and software) of the rover and understanding how each individual component came together to form the rover as a whole, and how this connected with the project theme. This included drawing out the designs for the rover and then 3D modeling those designs using Onshape; the components modeled were the rover chassis/case, sample picker and sample picker arms, and ESP32-Case. After creating the designs for the actual hardware, we first did our research into understanding how to create the system for moving the components in the rover. This was done through mentors, online resources, and attending club meetings. After research we gathered the necessary components and made the connections to get both the drive and sample picker system working. As for the software – controls and Anomalous Object Detection System – for the controls, we had to understand how to make a wireless connection to the rover and display controls over the web, to do that we consulted mentors and online resources. After that, using our own skills we created the wireless joystick and buttons. As for the LLM, we had to first decide on what LLM to use, after many decisions we decided to use Llama 4.1-Maverick developed by Meta which allowed us to integrate photos we took and detect them as anomalous objects. Overall, after creating each component on both the hardware and software end, we assembled it together. This was all done by us collaborating frequently on the project, meeting every weekend since the project's start to discuss where the project is going and what needs to be done. Outside of meetings, we discussed innovations and meeting times for the rover online.

Challenges we ran into

The challenges we ran into concern the Anomalous Object Detection System and the 3D printed materials. The problem regarding the Anomalous Object Detection System relates to the communication between the wireless camera and the constructed LLM. This issue stems from the need of the LLM to use the internet as it uses the Groq API to connect to the Llama 4.1-Maverick LLM, this is overridden when the wireless camera connects to the device that holds the LLM. To solve this we used a teammate’s laptop that can easily connect their phone screen to the monitor of the laptop through Screen Copy for Android and display the wireless camera via the phone, running the LLM and live feed simultaneously on the same screen. This works as the LLM and Wireless Camera are now on separate devices, meaning that the camera is now connected to the phone and not the camera, allowing for no interference. As for the issue regarding the 3D printed material, this issue arises from the fact that there were several print complications or limitations set on 3D printing by Makerspaces. This was eventually solved by dividing larger components and relocating where we did our 3D prints. A better solution would’ve been to use separate components that didn’t require us to 3D print, however, the main issue was solved in the end.

Accomplishments that we're proud of

What we are proud of is getting the wireless controls and camera to work as this was a large goal to accomplish, as we weren’t all too knowledgeable as to exactly how to go about this. This accomplishment was achieved with the determination, passion, team-work, and external aid of others. This was one of the main goals to reach in this project and it was a good thing we reached it. Another accomplishment was creating a vision model through AI, this allowed us to create a better mars rover as it does as it is intended to do, which was to survey an extraterrestrial planet and served as part of a learning experience.

What we learned

We learned about the vision models that were needed to create the LLM used in the project. That was a learning experience for all of us as we didn’t understand how many of those vision models worked, same goes for working with the wireless camera, and getting that to communicate with the vision model. Another learning experience stems from the web joysticks, it taught us a lot about languages we didn’t know about (mostly HTML and C) and how these joysticks integrate with the motor drivers on the ESP-32. As for the motor drivers and motors, we learned how an H-bridge works and its communication with the motors through voltages.

What's next for AI Powered Mars Rover Robot

For next steps, we would like to make the entire functioning of the rover more autonomous - the moving, the taking of images and putting it in the LLM, and picking and storing anomalous object. We would also like to add servo to make the turning smoother. We don't think the rover would be able to move very well in extremely rough terrain and accounting for that would also be essential for next steps.

Share this project:

Updates