Our team explored many potential project ideas a variety of different realms,
but nothing captured our attention like the Leap Motion. Leap is truly an 
innovation- a sensor the size of one finger can capture and map the motion
of all 10- and your hands, and your wrists. This allows people who don’t have
the acute motion control of all of their fingers to command all of the functionality
of a keyboard and mouse. But, the incredible capabilities of Leap Motion can be
appreciated by anybody, no matter who they are.
However, prior to this project, the Leap Motion capabilities were lacking in a major
way: there isn’t a lot of accessible material that incorporates artificial intelligence
into this revolutionary physical interface. We decided to explore how the user 
experience of Leap Motion games could be enhanced when incorporated with
an artificially intelligent opponent who learns alongside the user

What it does

At its core, our project is an interactive game, similar in intent to the
universally-loved game Codebreaker, now enhanced by an artificially intelligent
opponent and presented in an appealing interface that introduces the user to the
new capabilities of Leap Motion in a comfortable, 8-bit-inspired UI. 
The user and the AI-opponent face off in a “player-vs-player” viewpoint. The user
has access to four hand gestures, each mapping to one of the four elements:
Air, Fire, Earth, and Wind. Each has one goal: discover the most powerful
combination of these elements (in other words: a 4-action combination) in order
to master the elements before their opponent does and defeat them.
While the user becomes more comfortable with 4-action combinations, the
opponent learns along with them, and responds to an increase in player skill
by also becoming more skilled at the game. This creates an incentive for the
player to seek new and more powerful element combinations- the best of which
randomize at the beginning of each game- to win the game.
In addition, sounds for this game were all original. 
They were created using an electronic music editor, FL Studio.

How we built it

Or team is lucky to have a wide variety of talent, allowing us to
display strength at every level of game development. The front end was
created with Unity and incorporates several packages from the Leap Motion
sensor in order to map hand gestures into game moves that would have
traditionally been operated on a keyboard.The opponent was born and
raised in python and uses the keras tensorflow wrapper in its implementation.
The AI was coded and trained using reinforcement learning and was made
to wait for the user to output values and was also made to challenge each user
uniquely according to their specific level of improvement and gameplay.
The backend was also written in python and used Flask with json data
structures to interact with both the AI and the user input from the front end. 
In addition, another component was also added for the sake of truly
bringing the player’s reality into the game. We use twilio to interact with the
player by phone in a surprising and helpful way.


Naturally, undertaking a project like this was very challenging, and we faced
both predictable and unpredictable challenges. It is always difficult to
integrate the code written by four different minds into one seamless product. 
This difficulty is typically exacerbated by the time constraints of a Hackathon,
but by either some stroke of good fortune or good planning, our group ran
into relatively few challenges in this respect, especially considering how hard
 it could conceivably be to incorporate an artificially intelligent opponent- 
which needs information from both the front and the back ends- into a game.

Our greatest challenge, however, came with incorporating the Leap Motion
sensor. As this technology is relatively unfamiliar to most people (including
us- most of whom only just learned about the sensor’s existence a few days
ago), we had no prior experience incorporating it and thus had to learn on the
fly. Through the sheer dedication of our front end hacker, we were able to
get 3d hand models on the screen of a 2d game- already a monumental step. 
However, incorporating the hand motions took longer and was more
challenging than anticipated due to the lack of clear and complete documentation
of the Leap Motion sensor.


One of our great accomplishment was creating a game that features an
artificially intelligent opponent. Although the idea of having an artificially
intelligent opponent who has learned the game before playing against the
player is not new, it is rare to see a constantly-adapting opponent who
learns right alongside the user. This required a large interaction at all
levels but we are especially thankful to our team members who developed
the algorithm. This contribution creates a uniquely captivating experience
for the user and encourages them to improve and interact more with the game.

Our other great accomplishment was in the game interface development. 
Creating a fully playable game with coherent rules and structure is not trivial
because incentive structures have to keep players challenged without turning
them off from trying to improve. Our game handles this beautifully by having
a simple idea, enhanced by the intrigue of using the Leap Motion, balanced
by the challenge of beating a constantly-improving opponent.

All of us stretched our personal limits as well. 
Working with a team of strangers in a stressful environment to create
something coherent and beautiful in a laughably short amount of time
is inherently impressive- especially because we all had to learn new features
of programming languages in order to create such a robust final product.

What we learned

In the end, all of us have a lot to take away from the projects. 
We learned how to work together and come to a consensus on a
project that was both challenging and (mostly) possible to accomplish
in about a day. We learned how to keep our spirits up an keep grinding,
and how to laugh in the face of danger.

Also, at least one of us finally learned what “json” stood for- which
may or may not be an indicator of just how much we had to learn
in order to accomplish this project.

What is next

Part of responsible project design is creating minimum viable
products at multiple achievement levels. Being able to design a
complete project even with time and software constraints works
very nicely in a game, but there are plenty of opportunities for us
to make the game more fun and interesting to play.

In our original game design, we wanted to have multiple levels,
starting with just one hand gesture to introduce the player to the
game, and then in subsequent levels we would them how elements
could combine to become more or less powerful in the next level. 
We wanted to gently bringing them to the level where they had to
master 4 hand gestures, rather than throwing them into that in the
first place. In order to improve the game, we could add multiple levels
with higher and higher levels of difficulty.

With artificial intelligence, the sky is truly the limit. 
We can teach the AI to jump and crouch to dodge user attacks,
or do a variety of creative evasive or offensive maneuvers to keep
the player challenged and interested.

In addition, this idea can easily be envisioned in AR or VR in order
to truly capture the spirit of futuristic gaming brought about by the
Leap Motion.

Ultimately, the possibilities are endless. With an open-ended and
engaging game idea, combined with the technological prowess 
of Leap Motion and artificial intelligence, or game is truly one of a
kind and provides a unique user experience that cannot be found

In the game, we wanted to have reinforcements for the user such as
messages that say "Good Job!" or "Okay..." so that users could 
learn which combinations were more powerful than others.
However, we did not have time to implement this. The user can still
look at the damage they inflict on the opponent to see if they are
choosing good combinations, but for an improved game
experience, we could implement these feedback messages in the

Built With

Share this project:


posted an update

Huge milestone for E[Leap]ments - our DQN agent went from a standalone RL agent to a fully integrated in-game builtin AI! When a new user starts the game, the AI also starts from zero knowledge of the game and learns with you! Wanna see who learns the hidden rules faster? Come check out our project! :D

Log in or sign up for Devpost to join the conversation.