The Arm of R2D2

3D printer workstation
Rotatable base
Arm piece
Arduino connection to the gripper
YOLOv5 Object Recognition
Ultimaker Cura Model
openCV color and size recognition

Inspiration

Inspired by R2D2, our team crafted a robotic arm in 24 hours. Since R2D2 only moves on wheels and has a 2-D moving arm, it reminded us of people with accessibility issues. Therefore, we wanted to extend upon the design and create a hack that helps people who struggle to grip, grab and move items such as patients with arthritis or elderly people so that they can live comfortably at home. This project incorporates sophisticated 3-D printing, advanced computer vision and object detection, wireless data transformation and motion localization. This project reflects the endless possibilities of artificial intelligence and robotics to improve people's quality of life and help those in need.

What it does

Our robotics arm features a sophisticated suite of capabilities, which uses 3-D printing technology, OpenCV and YOLOv5 object detection, wireless data transformation through Python and Bluetooth (Internet of Things) and motion localization to achieve. Our hack hopes to help people with accessibility issues such as an older population or patients with arthritis. Our arm uses AI recognition to locate, grab and move specified items for people in need. The arm adeptly identifies objects by dimensions, colour and name through trained AI software with YOLOv5 and provides accurate coordinates which our arm uses to navigate to it. The computer processes the raw information from the camera, translates it into precise movement controls through a process derivative of forward and inverse kinematics and sends it to the arm through a seamless Bluetooth connection. Notably, our robot has endless possibilities as you can even incorporate other custom object detection software with YOLOv5.

How we built it

In this robot, we used a variety of different technologies and skills such as 3-D printing, OpenCV, YOLOv5, Wireless Data Transformation with Python and Arduino and motion localization.

Computer Vision: Using OpenCV, we were able to categorize objects based on size and colour and return an accurate location of the object. With “cv2.findContours”, we were able to correctly identify the contours of the object which were used to determine the size and location and the centroid of the object. With the “cv2.cvtColor”, we could also use it to identify objects based on colour. On the other hand, we also used YOLOv5 to categorize objects by name through pre-trained AI object recognition software. Using “torch.hub.load('ultralytics/yolov5:v6.0', 'yolov5s')”, we could access the YOLOv5 Open Source library and use its AI directly.

Wireless Data Transformation and Motion Localization: Using an Arduino Bluetooth Module, we were able to connect seamlessly to a computer where we processed the data. By using the computer vision data, we processed using an onboard script on a computer which sends movement commands via Bluetooth to the Arduino Mega board that controls the arm—using a serial connection such as the function serial.Serial('COM5', 9600), we were able to establish a wireless, high-speed connection with the board and send the board anything we wanted.

Mobile User Interface for Arm Control: We crafted a mobile application utilizing MIT App Inventor, facilitating direct communication between a smartphone and a robotic arm. This innovative application leverages MIT App Inventor's block programming capability to capture user inputs via a specially designed interface, which then translates into commands for our Arduino-based control system.

Our robotic arm is driven by five servo motors, each governing the movement of the arm's joints with a rotation range of -30 to 180 degrees. The application features adjustable slider bars on its interface, enabling users to manually fine-tune the rotation of each joint.

Additionally, our system is equipped with the capability to record and store sequences of movements. This allows users to create and replay complex motion patterns with ease, simply by interacting with the "save" and "run" buttons on the app’s interface.

Challenges we ran into

Throughout the hackathon, we ran into multiple different challenges. First, we were challenged when first determining the viability of our plan. Because of the sophistication of the arm, it required a predicted 3-D printing time of 50 hours, which seemed impossible. However, to navigate through this problem, we set up three 3-D printers in parallel, which were able to print the whole arm in just under 20 hours.

The second challenge we ran into was the computer vision implementation. We struggled with the camera's live feed transmission, frame rate, accuracy of object detection, and focus of the video. Our team effectively bypassed this problem by debugging code, modifying object detection parameters and adjusting the port of the camera module.

Accomplishments that we're proud of

Even though this is our team’s second Hackathon, we were ambitious. Even though this is just a proof of concept, we were proud of the fact that we were able to implement computer vision, assemble and learn how to 3-D print and program the arm to move all within 24 hours. Therefore, we are proud of the progress our team made in the last 24 hours despite having little to no experience in most of the aspects of the project.

What we learned

Throughout the hackathon, we learned many essential skills of this project. For instance, we were able to assemble three 3-D printers and learn how to slice model and print effectively the structure of our arm within 24 hours. Furthermore, we also learned how to implement our ideas into reality. Since our group is composed of all novices, we had little prior experience. However, we were able to learn how to implement OpenCV, YOLOv5, Bluetooth data transmission, motion localizing, app development and 3-D printing all in 24 hours. Lastly, we learned the joy of participating in a hackathon together, the teamwork needed and the challenges we would face in this scenario.

What's next for The Arm of R2D2

Since The Arm of R2D2 is only a proof of concept with many parts unfinished, we wish to polish the final product and be able to share it with the world. By fine-tuning computer vision, completing the integration of computer vision and robotic motion and adding more axis of motion would be our next steps. We want The Arm of R2D2 to not only be a proof of concept, but a vision of our bright future, not only with smart homes but in all aspects of our lives.