ClearPath

ClearPath

Inspiration

We thought this was an innovative idea that would work really well the Overshoot API, ElevenLabs text-to-speech, and WisprFlow's speech-to-text tools. There are 1 million legally blind people in the US, with 7-8 million being visually impaired. ClearPath aims to solve that problem and allow everyone to find where they need to go.

What it does

ClearPath is a real-time, voice-controlled navigation system that adapts continuously to a user’s surroundings. The user initiates navigation through WisprFlow speech-to-text, which captures intent and context, after which live camera footage is streamed via WebCameraView to the Overshoot API for ongoing environmental analysis. This visual data is then combined with Gemini’s scene understanding and pathfinding logic, and a navigation engine merges both streams to generate clear, step-by-step guidance. Instructions are delivered through natural speech, while Overshoot keeps analyzing new signals in the background. Users can move through the experience by saying commands like “next,” “repeat,” or “previous,” which trigger fresh camera analysis and refined guidance, creating a responsive loop that adjusts in real time to both the environment and the user’s requests.

How we built it

ClearPath was built using React Native and Expo, enabling cross-platform deployment to both iOS and Android through a single TypeScript codebase. We integrated WisprFlow's speech-to-text capabilities with the Web Speech API to capture voice commands, which trigger the WebCameraView component to capture real-time frames from the device camera. These base64-encoded frames are sent simultaneously to both the Overshoot AI API for object detection and obstacle analysis, and to Google Gemini's Vision API for deeper scene understanding and pathfinding logic. Our custom navigation engine merges these AI responses into concise, actionable directions, which are then converted to natural speech using ElevenLabs' text-to-speech API with selectable voice personas. The entire application runs through an Expo development server with ngrok tunneling, allowing secure HTTPS access for testing on physical devices. We designed the system as a continuous feedback loop: each voice command or environmental change triggers fresh analysis, ensuring guidance stays accurate and responsive as users navigate indoor spaces.

Challenges we ran into

The idea was to originally integrate this into a Native IOS app for mobile phone usage. With a 24 hour deadline, we did not have enough time to build out a fully deployed mobile app. We then resorted to using ExpoGo via the iPhone, though Overshoot SDK is not compatible with ExpoGo and can only work on PWAs (Progessive Web Apps). This significantly impacted our project as we had to pivot 10 hours in when we called quits after hours of unsuccessful debugging. We also struggled a lot with Overshoot due to the WiFi/Hotspot issues, though everyone had the same experience there. Another huge issue was that we had no access to an Indoor Mapping Plan. Due to security/privacy data, many organizations/buildings do not release this data. This made Overshoot much less effective than we had hoped as it had a lack of indoor mapping data to refer to (on scale). Lastly, we wanted to import a HuggingFace model to help with indoor navigation, though due to time constraints this also did not go to plan.

Accomplishments that we're proud of

Our team is proud that we saw the hackathon through despite running into many barriers. We had a great time building this, albeit frustrating running into errors. With Emaad + Moeez from Canada, and Saanvi (Maryland) + Naveed (New Jersey) from the US, it was awesome to compare college experiences from each country and we had lots of fun talking amongst ourselves through this process. We are also proud that we were able to integrate many NEW technologies (Overshoot SDK, ElevenLabs, WisprFlow) even if it was not at the degree we originally hoped to achieve.

What we learned

We learned a few things: 1) We should have tested Overshoot SDK on mobile apps before hand. This would have saved us lots of time and made our action plan much easier. Obviously with newer technologies, there are bound to be limitations, and that is not the fault of the Overshoot team. 2) We needed to spend more time picking ideas. While we were all busy with work and did not get to thinking about ideas seriously until late Friday night/early Saturday morning, we would have made much more progress/a structured plan had we begun hacking at 1pm instead of 3:30pm when we finalized our ideas. We think this is due to the nature of this hackathon as there are many sponsors, and once we saw them/read into them, a lot of our original ideas changed.

What's next for ClearPath

Indoor Mapping plans are the biggest bottleneck to creating services like this. Had we had access to a Google/Apple maps API that contained Indoor Mapping data, we would have been able to leverage Overshoot in a completely different manner and made the technology much more effective. To get an application like this working really well at scale, this data needs to be available and its one issue that Apple and Google have faced for years. And in order to take ClearPath to the next level, we would need to find a solution to generated this data accurately.