We were inspired by the idea of performing surgeries online in remote areas where there may be a limited amount of doctors or difficult access.

What it does

Remote Handy Craftsman uses a robotic hand that can be controlled by a motion sensor. The remote environment the hand is to be used in can be seen through an online web camera. Facial recognition and facial movement detection is used to effectively position the camera relative to the target.

How we built it

It is built with Python Flask, Flask cors, Ngrok, and Servos. The hand is controlled with Leap motion sensors and programmed using Arduino Serial Interface with Python, and a mixture of C++ and Java. The web applications including the browser embedded video calling and facial feature detection were created using HTML, CSS, and Javascript. We also included a customized 3d-printing using SolidWorks to create rotating phone stand that rotates in relation to face detection. All of our web applications were deployed using Microsoft Azure.

Challenges we ran into

Our project was full of challenges but we overcame all of them through hard work, collaboration, and innovative thinking. Connecting all the components was difficult as well as finding a solution for problems such as the detectable range of motion for facial recognition, phone rotation using Lego Mindstorms, and using a DragonBoard.

Accomplishments that we're proud of

Successfully being able to get the face detection to work and deploying all the components of our project using Microsoft Azure.

What we learned

We learned how to integrate IoT into our project, how to use Microsoft Azure, and how to use facial recognition APIs.

What's next for Remote Handy Craftsman

Improvements in connectivity and user interface.

Share this project: