-
-
Fullscreen UI that can be toggled with a button in the top right corner.
-
Example of a user path that nearly ties with Gemini's most optimal path
-
Hover path drawing using your finger!
-
Statistics from a failed user path and the corrected optimal path created by Gemini.
-
Path Correction when the user's path crashes into the cloud obstacle
-
Demonstration of Gemini's most optimal path vs your original path with included statistics below the interface.
What Inspired Us
Our inspiration comes from the theme of the year's HackRPI, "In the Clouds". We hope to connect computer vision, artificial intelligence, and physical simulation to create a project that is both interesting and challenging. Therefore, we developed Nimbus, a simulator game where players can control the drone with their hand gestures. The player only needs to draw a flight path with their fingers in front of the camera, and the drone will attempt to fly along this path while also trying to avoid the effects from clouds and winds.
How We Built Our Project
To make the game more challenging, we also incorporated Gemini AI, which allows players to compare flight routes and completion times with the AI. In terms of technology, we used Python, JavaScript, HTML, and CSS to build to this project. We also utilized MediaPipe to recognize gestures, and processed game logic, route calculations, collision detection, and physics simulation through Python. We also generated candidate routes using the Gemini API and then tested these routes using a Python simulator to select the fastest and most reliable solution that could reach the destination.
Challenges Faced
During the development process, our biggest challenge was to integrate the front-end, back-end, and AI functions within just 24 hours while ensuring the smooth operation of the game. Through this HackRPI, we not only learned how to use computer vision and AI APIs, but also gained a deeper understanding of teamwork, collaboration, GitHub management, and how to complete a truly functional software project from scratch.
