Inspiration
We wanted to create a project that stayed true to the theme of arcade games and implemented augmented reality to expand our skillset. Why go to the arcade when you can take the arcade with you?
What it does
We take in a video stream from the camera on one's personal device, then output back to the user a view where they can see the hitboxes where they need to move and interact in order to activate a certain channel. When an arrow approaches the screen, on cue with the beat of the music, the player must interact with the appropriate hitbox in order to score points. We are also able to easily convert songs that are already in the rhythm game OSU into songs that are playable in our game, so there are thousands of potential maps.
How we built it
We built the application by splitting different groups into tasks. We had one member focus on the style, webpage, assets, and design of the application, while another group focused on implementing the object detection system, and a final member focused on implementing three.js and the system of having the arrows "fall down" to the player while also having hit detection.
Challenges we ran into
We ran into several optimization issues in detecting when a player's foot entered the appropriate area. We had to implement a background subtraction method that then uses a contour detection system and cross references with the ratio of white to black pixels. Additionally, we had never used three.js before so there was a large learning curve in trying to create the game area and display it correctly. I ended up having to spend a lot of time figuring out the math of moving objects around when there were probably easier ways of doing things.
Accomplishments that we're proud of
Organizing ourselves in a well-coordinated team environment. We were exceptionally proud of being able to merge the several features altogether. Likewise, we are proud of building together a project that interacts with the real world as it is our first implementation of such. Additionally, being able to utilize newly learned material for our urgent use cases.
What we learned
We had the opportunity to learn and explore OpenCV and object detection methods as well as create an interactive webpage that utilizes 3d models using three.js We learned a lot about how to effectively use third-party libraries, get them to interact with each other to achieve our goal, and get them to build correctly (some struggles there).
What's next for DDR-AR
We really wanted to expand augmented reality and implementation. We hope to implement a version where the arrows are streamed directly to the video stream to the user, rather than just the buttons being present.
Log in or sign up for Devpost to join the conversation.