Robo-view

The Arm!

Inspiration

Many advertisements of smaller products like smartphones, wearable devices, foods, and more showcase in dynamic angles to better appeal to viewers. Seeing how the SO-101 robotic arm moves around with its attached camera reminded us of this, and gave us the idea to replicate that cinematography using algorithms, making the process more streamlined, smooth, and efficient.

What it does

There are two parts to the project, the app and the arm. The app has voice and text input for users to command the arm, and then after the arm moves around and records at various angles, it sends the video back to the user on the app. The arm then is the second part, and it rotates and traverses around any product using 1 of 3 different dynamic movements as it records.

How we built it

We split into different groups to work on operating the arms, creating the software, and handling the machine learning. After teleoperating the follower arm to the leader arm, we ran trials to replicate movements and recording angles for cinematography, analyzing the motor arm data and then sending it into the Isaac 0.5 model to train on through ML. We recorded 25 different tests per movement, resulting in around 75 tests total. Through that AI model, we then replicated the arm movements to account for different recording angles explained from user audio input put into the app, and also different object positioning.

Challenges we ran into

The main challenge we ran in to was after doing all the recording and training was relabeling all the data with bounding boxes, which took a bit of time to do while navigating the Perceptron API.

Accomplishments that we're proud of

Setting the arm up, training it, getting it to run successfully after hours of work.

What we learned

We learned various skills such as robot manipulation, combining agents and tool-calling with the physical world, and creating software for video editing as well.

What's next for Robo-view

We think it's best to give the robot more movements, and training with arms that have more degrees of freedom. This could allow for my dynamic shots with greater ranges. We'd also like to train it so that it can identify and record a single object within an environment of other objects.