Virtual Thing

Inspiration

We initially thought of something that could blend both UI/frontend design along with some sort of hardware or interactive component. We ended up with an idea where the user can grab objects in a virtual 3D environment by just moving their hands, and can move around in this environment.

We thought that this could be used, for example, to help people with hand injuries work on their range of motion during physical therapy, and could be extended in the future with games to motivate them, or to expand the availability of situations in the environment to simulate surgeries or bomb defusals for example.

What it does

Using the camera, it will recognize hands (up to 2) and using them you can move around and look around in the 3D environment. You can also pick up and drop items.

How we built it

We used a ML computer vision library by Google called MediaPipe, along with a 3D frontend library Three.js and NextJs for the rest of the webapp.

Challenges we ran into

The biggest challenge was getting the motion and rotation to work. We initially tried a method where you could choose objects by making a number with your left hand, could move the left hand to rotate the camera and use the right hand to move around and pick up objects.

However, we learned that MediaPipe's depth perception was very inconsistent, and moreso in general single-camera computer vision applications are not very good at precise depth perception, so it made the movement very bad. However, on doing some research we incorporated elements derived from Guiard's Kinematic Chain Theory and Theory of Bimanual Control, we were able to make a much smoother, more coherent, and more enjoyable experience.

Accomplishments that we're proud of

We were really proud of figuring out the right control scheme to make the application work really well.

What we learned

We learned a lot about computer vision, as well as how it pertains to machine learning and object/gesture classification. Also, we learned a ton about human-computer interaction and Guiard's Theory of Bimanual Control.

What's next for Virtual Thing

We believe that this project has a lot of potential for future use and expansion. As we already discussed, it could be used to help injured folks work on their range of motion during physical therapy, or to help people train for precise but dangerous tasks like surgery of bomb defusal.