With the introduction of CoreML and Vision into iOS11, we thought it would be a great opportunity to use these new tools to create a new type of game- one that could utilize the greater potential of iOS.
What it does
Pong Vision is a game of pong where your paddle is controlled by your hand. After calibrating the vision, moving your hand left and right across the camera's view will cause the paddle to also move left and right on the screen of the phone. Currently, the other paddle is controlled by a simple movement script- however, in the future, people will be able to play from multiple devices via matchmaking.
How we built it
We built most of the code for the actual game, just from coding in Swift. For the object recognition, we found a repo on Github that had some basic object recognition using Vision, and so we used the code and the tutorial to build and customize the recognition portion of the app. AWS was planned to be used to sync devices together, allowing real-time matches to take place. For that, the Mobile Hub SDKs were used. DynamoDB and Cognito had some part in the development process- however, they ultimately failed, and we had to scrap them.
Challenges we ran into
We both worked on different parts of the app, so we encountered different challenges. Caleb worked mostly on the game and Vision portions of the app. He encountered some major problems- namely, the inability to write proper objects and utilize SpriteKit. As a result, the game itself took much longer than expected to build, and the vision was much more inconsistent. Charlie worked on the AWS part of the app. He also encountered several serious challenges- namely, Swift not playing well with AWS. The most major problems were on saving- he was unable to find a good integration of AWS with Swift, and with little good documentation, he was unable to finish most of the AWS services.
Accomplishments that we're proud of
We figured out Vision well- the hand syncs up with the paddle nicely, and the game itself plays relatively smoothly. In addition, we had never used AWS - it was a challenge to try to learn it, but we did make code that
What we learned
We learned how to use parts of CoreML and Vision from iOS kits. We also learned how to use AWS, and how to integrate it into our apps.
What's next for Pong Vision
We plan to continue integrating true multiplayer- the hand motions were really cool to use, and we feel that further integration with AWS and good multiplayer gameplay would greatly enhance the game and the experience.