An American traveler took a trip to Thailand, one of the most stunning places he visited. He was enjoying the landscape and food. But one awkward thing happened, he didn't understand nor speak Thai. The owner of the restaurant seemed a bit unhappy when he gave him a thumb up gesture. Not knowing why the owner got unfriendly all of the sudden, he left the establishment anxious and confused.

What it does

Defining your native country, recognizing commonly accepted gestures, identifying countries that would find that common gestures offensive. GestureAR is designed to fulfill the educational needs of teaching and informing novice travelers appropriate and inappropriate gestures. Through that way, people might share more respect for the different countries and be prepared for the incoming international journeys.

How we built it

We used HoloLens2, Unity and MRTK to build the whole environment. The most crucial part is using AI to train the machine learning model in the cloud to identify different gestures.

Challenges we ran into

We have many technical issues along the way, for example, we have met a GitHub Repo data quota when our all files had been locked and we did not have the access to push or merge anything to our base project anymore.

Accomplishments that we're proud of

Trained AI model to the machine learning model in the cloud to identify different gesture and then we inform users with multiple gestures in different cultures.

What we learned

How to train AI to identify hand gestures with small amount of data.

What's next for gestureAR

Ultimately, this AR application will expand your Metaverse cultural identity so that every culture could be represented.

Share this project: