AeroPitch_Team_Froggy

Here is the story of your project formatted for your submission.

Inspiration

The project was born from a desire to address the navigation challenges faced by visually impaired individuals in unfamiliar indoor environments. While traditional aids like white canes and guide dogs are essential, they are fundamentally limited to physical touch or immediate proximity. This inspired the creation of a flying eye prototype—a drone companion that leverages spatial AI to perceive the world from an elevated perspective and communicate environmental data through natural, conversational dialogue. The goal was to transform a standard unmanned aerial vehicle into an empathetic assistant capable of seeing what the user cannot and explaining it in real-time.

What it does

AeroPitch_Team_Froggy is an intelligent drone simulation that acts as a hands-free guide for the blind. The drone stays in a grounded state until it hears a voice greeting or a command to assist. Once active, it uses a suite of sensors and AI logic to identify furniture, doors, and obstacles within a room. It speaks back to the user to confirm commands and provides real-time warnings if the path is blocked. By combining voice interaction with autonomous spatial awareness, it provides users with a safe, navigable "mental map" of their surroundings.

How we built it

The technical architecture was built using a multi-layered approach centered around the Python ecosystem. I developed a custom 2D simulation environment using the Pygame library to model the physics of indoor navigation and obstacle avoidance. For the vision component, I integrated the Ultralytics YOLOv8 object detection model to identify landmarks. To create the brain of the companion, I connected the system to the GitHub Models API, utilizing large language models to process user intent and generate human-like responses. To ensure safe navigation, I implemented proximity sensors based on the Euclidean distance formula. The interface was finalized using the SpeechRecognition library for audio input and the pyttsx3 engine for text-to-speech output.

Challenges we ran into

One of the most persistent challenges was managing the technical constraints of the Windows environment. Installing heavy machine learning libraries like Torch led to critical errors due to the maximum path length limitation. I resolved this by restructuring the project directory into a root-level folder and utilizing a virtual environment to shorten file paths. Furthermore, I had to overcome the blocking nature of speech recognition, which threatened to freeze the simulation. I addressed this by implementing multithreading, allowing the Pygame rendering loop to maintain a consistent frame rate while a background thread handled the heavy processing of voice capture and AI inference.

Accomplishments that we're proud of

I am particularly proud of achieving a seamless loop between voice input, AI reasoning, and physical simulation. Seeing the drone respond to a natural phrase like "Hello, I am lost" by taking off and identifying an exit was a major milestone. Successfully integrating a high-level Large Language Model to act as the "pilot" allowed the project to feel more like a companion than a piece of software. Overcoming the deep-rooted OS path errors and threading issues also represented a significant personal victory in technical troubleshooting.

What we learned

Through this process, I gained a deep understanding of the intersection between spatial AI and accessible design. I learned that creating a tool for the visually impaired requires a focus on timing and clarity; the latency between detection and speech must be minimal for the information to be useful. I successfully navigated the complexities of connecting cloud-based AI models to local hardware simulations and mastered the management of sophisticated Python environments. This project taught me that the most impactful technology is not just about raw power, but about how effectively it can communicate and assist in the real world.

What's next for Team_Froggy

The next phase for Team Froggy involves moving the simulation from a 2D environment into a 3D digital twin to better model complex vertical obstacles like hanging lights or shelves. I also plan to implement edge-computing solutions to reduce the reliance on a constant internet connection for the AI brain, ensuring the drone can function in areas with poor connectivity. Finally, I hope to integrate haptic feedback into a wearable device so the drone can "nudge" the user in the right direction alongside the voice guidance.