Ghost Hands

Live rendering of Rubicks Cube (Paul Cube)
Key Notes of Rubicks Cube for AR Learning Model (Paul Cube)
AR Simulation of Rubicks Cube (Paul Cube)
Live rendering of ASL (ASL)
Key Notes of ASL (ASL)
AR Simulation of ASL Training (ASL)

Inspiration

Beginning to plan our project, we knew one thing from the very start- we wanted to make more than just something on the screen. Specifically, we wanted to implement some sort of AR/VR to be able to really help give effect. With some influence from a team member who had a recently rough time with training due to a lacking professional, we came up with the idea of ghost hands- a software that allows you to receive the training of a professional wherever you are.

What it does

Ghost hands takes in video input and scans the hand input along with the objects they touch. Using this data, the Meta Quest 3S our project uses generates a set of translucent ghost-like hands that mimic the actions of the inputted video. It then gives training steps overlayed over a real world environment on top of the headset, showing the user how to perform the task, and giving them a rating linked to how close there movements where with the designated task.

How we built it

Our Project is split into two main facets- half of it being a build of mediapipe to scan videos for hand movements, along with a database connection connected to a frontend webpage that allows for video storage to be transitioned and scanned to allow for training. The other half is the fun part- it consists of the Meta Quest 3S, which takes the inputted data that's stored after going through media pipe, and uses it to make the Ghost Hands. We then used the Meta Quests hand tracking capabilities to get data of the users hands, and compare that to the training data.

Challenges we ran into

For challenges well.... we ran into quite a few. One of the big ones was getting our initial pipeline set up between the database and media pipe. Though it went smoothly getting our endpoints set up along with getting the video to go through the initial steps, tuning it to properly take in the hand movements was a fairly large challenge at times. Additionally, we ran into a lot of issues setting up our initial environment with the headset, as though some of our members had experience with unity, setting it up to work with the headset was quite difficult at times.

Accomplishments that we're proud of

Really, I'd say what we are proud of the most is being able to set up and have our full pipeline working along with the VR. It was extremely rewarding to be able to see our idea come to life.

Additionally, we had a sub 5 second analyzation time for the mediapipe scanning of the hands, which took a lot of work when it came to honing the process.

What we learned

Really, we all learned a lot- though it boils to experiencing more how to properly set up a dev environment within our team. Additionally, working with the VR headset allowed us to pick up a lot more on how unity works, along with connecting it to outside services via a database.

What's next for Ghost Hands

Quite a lot really- next up, we have plans to extend ghost hands into more visualizing the video through generated images of the objects being touched, rather than basic matching. Additionally, we want to further help learners by giving more detailed descriptions, hoping to be able to further develop the learning process while going through a ghost hands tutorial.