SignSense

Inspiration

We wanted to create a game that challenges players' 5 senses and it's importance in our daily lives. However, it also challenges the human's capability to adapt to situations while losing the sense of hearing. This challenges us as a group to find other ways to communicate such as ASL and visual awareness increases. Although, in a video game, the situations aren't very realistic but it keeps the human mind on their toes. By using motion sensors and Unity game engine's basic tool kit, we allowed players to access the ASL community by signing key words to execute game play commands as well as circumventing the language barrier between ASL and Non-ASL users.

What it does

The project accepts motions and signed words in front of the camera, interpreting it using our own data set, and performing a specific action related to that sign. The program executes the game action and displays what word or key words were signed by the user. Leap Motion Camera detects the motion and corrects the user to the correct signed motion.

How we built it

Unity game engine to make the environment and using the leap motion camera to receive motion control input from players.

Challenges we ran into

Learning how to use Unity game engine and incorporating leap motion camera technology and utilizing its libraries and example assets.

Accomplishments that we're proud of

Navigating through unity, creating our own data set of gestures and hand poses, and making our own scrips in C++.

What we learned

How to use the different assets of a game in unity, creating our own assets and the scripts for them. How to include addon for Unity which was the Leap Motion Camera toolkit.

What's next for SignSense

We are eager to improve the data set and its organization to increase the amount of words and commands to the game. Improving the physic interactions with the player and it's environment. Researching the Camera hardware and its tool kit to improve the user input. Adding customizable characters and abilities to increase capability to learn Signed Language.