Remote Handy Craftsman

Inspiration

We were inspired by the idea of performing surgeries online in remote areas where there may be a limited amount of doctors or difficult access. But knowing how difficult surgeries would be, we just wanted to build a prototype robotic hand to be used in hostile environments where humans cannot operate (lava fields, nuclear power plants, etc).

What it does

Remote Handy Craftsman uses a robotic hand that can be controlled by computer vision algorithms. The remote environment the hand is to be used and can be seen through an online web camera. Facial recognition and facial movement detection is used to effectively position the camera relative to the target.

How we built it

It is built with Python Tornado API (websocket), MediaPipe, Ngrok, and Servos. The hand is controlled with MediaPipe's computer vision library and programmed using Arduino Serial Interface with Python, and a mixture of C++ and Java. The web applications including the browser embedded video calling and facial feature detection were created using HTML, CSS, and Javascript. We also included a customized 3d-printing using SolidWorks to create rotating phone stand that rotates in relation to face detection. All of our web applications were deployed using Microsoft Azure.

Challenges we ran into

Our project was full of challenges but we overcame all of them through hard work, collaboration, and innovative thinking. Connecting all the components was difficult as well as finding a solution for problems such as the detectable range of motion for facial recognition, phone rotation using Lego Mindstorms, and using a DragonBoard.