OmniGlove

Picture of the humanoid robotic arm end effector being tension and tuned by Ambarish.
Picture of the controller glove being wired and tested to confirm the sensor output by Avanish.
Picture of the camera setup calibration and creating a protocol to calibrate it anywhere. Mehmet is also testing the glove with the setup.
Picture of using the controller glove to directly control the humanoid robotic arm end effector, where each finger is linked to the other.
Picture of the overall assembly put together on the LeRobot arm with the camera setup in the background, can be calibrated to any location.

Inspiration

The inspiration behind our project was first finding concepts we wanted to learn from, in particular April Tags, IMU, and overall project integration skills. We decided on making our project to control the follower arm using a human hand as the leader by looking at what hardware we had access too, in particular the hardware AMD provided with the LeRobot Kits. Our idea was to make a general universal control interface to control robotic hands with know inverse kinematics for XYZ position.

What it does

Our project maps the motion of a human hand to the motion of a robotic glove by tracking XYZ location using a locator beacon and orientation using an IMU on a control glove. This is seen with a camera setup which is easily calibrated with a protocol that takes ~1 minute to do. The glove then is able to control the arm, where if you move the glove a certain way the arm moves that same way to better imitate the motion of a human hand. This can be used to control robotic arms in a much easier and more intuitive way, speeding up the implementation of AI robotic arms.

How we built it

We built it with 3 main components, the motion capture glove, the humanoid robotic hand, and the arm manipulation software. The motion capture glove using an ESP32 to communicate data from sensors like an IMU to other components. The humanoid robotic hand is meant to act as a better and more intuitive end effector. The software is meant to integrate the whole project together.

Challenges we ran into

We ran into challenges with our robot arm towards the end of the hackathon where all the motors on the arm we were using bricked and no longer connected. We had to get a new arm and recalibrate it with just 1 hour before the DevPost submission, but managed to get it back up to speed but unfortunately ran out of time for reimplementing our humanoid hand onto the new arm.

Accomplishments that we're proud of

We are proud of all the different projects we made and what we learned while making them. Each component of our project was technically challenging in a different way, but we learned, adjusted, pushed through and had fun.

What we learned

We learned a lot about sensors and how to actually implement them together into a large project. In particular we learned a lot about the IMU calibration and certain techniques we could use to help mitigate IMU drift such as using the accelerometer to somewhat compensate for drift in the IMU. We also learned a lot on the software side, particularly with system architecture and organization.

What's next for OmniGlove

We plan to better compact the hardware, get the humanoid robotic arm back up to speed and fully reliable, and to add soft limits in the software to protect the hardware if t it moves to a location out of bounds.