Inspired by our love of games and the interest in learning new ways to interact with the world, we threaded together several different systems, both programming and hardware, to make Dungeon Crawler. Ultimately we would like to see this develop into a full fledged experience for players, allowing them explore familiar spaces in new and fantastic ways using the largely unexplored world of Augmented Reality.
What it does
Dungeon Crawler is an augmented reality experience that has a robot navigating the space around it and you, strewn with AR tags, and visualized to create a virtual environment to explore in a familiar space. Players can control the robot using standard gaming keys. The robot can move precisely in response, controlled by an Arduino and Raspberry Pi. Finally the robots webcam picks up on AR tags which are converted into game object visuals in Unity.
How we built it
Because we didn't want to focus on the machining aspect of robotics, we started with a base robot but quickly built out from there software-wise. First we got the robot moving with an Arduino micro controller, using basic PWM to vary motor speeds. We then integrated a Raspberry Pi to allow for greater onboard computation, primarily the use of OpenCV, to allow for image and AR tag recognition, as well as wireless communication with a laptop. On the PC side we built Unity functions to take in a stream of images sent from the Raspberry Pi. Any AR tags recognized are converted visually to game objects. Any keystrokes are then sent back to the Raspberry Pi and Arduino to move the robot.
Challenges we ran into
Our largest challenge was communication between devices. Using 4 different powered devices, three of which all needed to run different code (Arudino-Arduino C, Raspberry Pi - python, Laptop - Unity C#) while still being able to transmit usable information to one another. Of them, the most difficult was getting the Raspberry Pi's webcam stream to Unity for the player to see. Sending so much information proved to be difficult, with us trying both the raw camera feed as well as sending still frames.
Accomplishments that we're proud of
Our proudest accomplishment of the project was solving so many diverse problems. From hardware with the devices, to information collection on the webcam to the back end computation to recognize AR tags and process images, we were really glad to see this come out as both a hardware and software hack.
What we learned
We learned much more about each of the individual devices (Arduino, Raspberry Pi) as well as how to communicate between them. Coming in with a very diverse set of skills, we also learned a lot from each other in areas of programming and debugging.
We also learned the importance of scoping a project appropriately. There were a lot of modules to this project and because of that many unrelated debugging issues came up, which took a lot of time to solve, especially because of our limited knowledge in some areas.
What's next for Dungeon Crawler
Going forward we would like to get better picture and video quality to Unity. We would also like to vary the kinds of game objects generated over the AR tags to take on a more realistic and adventure-like feel.