WanderWise

Our app provides a catered audio-tour-like experience for the world. Hear interesting and enriching information while you explore significant sites and hidden stories anywhere you are.

Comment

Inspiration

Every day, we walk or drive right by hidden treasures. WanderWise provides a hands-free experience to discover and learn about the world around you, anywhere you are.

What it does

WanderWise is a mobile and web application designed to provide users with a personalized and immersive audio tour experience anywhere in the world. The app utilizes online data and leverages the power of ChatGPT and ElevenLabs to generate voiced audio tour-like descriptions of points of interest. Simply go to a location you want a tour of, press the search button, and start exploring all of those interesting locations in that area.

How We Built the Project

  1. Frontend and Backend Development: We used Flutter and dart for both the web and mobile platforms. We hooked that interface into Google's API for mapping and pin placement. The backend was developed using Python for prototyping and Go (Golang) at the end for its better threading capabilities. We used BadgerDB to store generated data to provide quick responses for already generated points of interest. We then hosted on Digital Ocean.

  2. LLM and Audio Generation: We utilized the power of ChatGPT to generate audio tour-like descriptions and in the future would also filter locations out by user preference. ElevenLabs is then used to produce a realistic sounding audio tour file that we play when you interact with a map pin. ChatGPT and ElevenLabs integration is done in the background for each location and populated as soon as it's available.

  3. Data Fetching: We integrated Wikipedia information and locations to get data to feed into ChatGPT. We focused on interesting tidbits that would be found in a good audio tour. We also fetch images and other relevant data through Wikipedia and could add more sources in the future.

Challenges Faced

  1. Data Quality and Availability: It's hard to gather multiple sites of information and correlate it to a single source. Additionally, some websites don't have easy APIs and we may need to parse websites for text ourselves. To reduce this hurdle, we focused on Wikipedia as our first data source as it has a good API and interface to fetch data from. In the future, we can also add Google and many other crawlers for extra data.

  2. Prompt Engineering: Getting ChatGPT to properly summarize entire documents worth of data into an audio tour experience required a bit of effort. Sometimes the generation would be too long, or go slightly out of context. Sometimes one location would be great, but another would run into some completely different issues in its summary.

  3. Interface Responsiveness: We needed to make sure the app was responsive and not hanging while we wait for ChatGPT and ElevenLabs to produce results for nearby points of interest. To do this, we had to split up some of our backend responses to give data as soon as it was available and integrate a database to quickly cache and send data for sites that you've already visited.

Accomplishments that we're proud of

  1. The appearance of the frontend interface. Corlin on our team was able to quickly prototype this and got a functioning demo very early so we could polish it as we continued working.

  2. Mid-development language transition. We pivoted from Python to Go over half way through development to ensure we provide a better user experience and avoid network delays. This was challenging and posed a lot of risk of introducing new bugs.

  3. How effective our app was at providing interesting new information. We found a couple of sites out at sea that we would never have known about otherwise, simply because our app pointed them out. A sunken destroyer and a growing reef.

What We Learned

  1. Project segmentation to accomplish separate prototypes quickly. In our case, our front end, server link, and data fetching were all broken up as separate tasks to tackle at the same time.

  2. Speed and responsiveness is key to keeping users engaged. We switched from Python to Go in the middle of development to ensure we could make faster responses to the frontend client and get back those important locations and descriptions as soon as they were available.

What's next for WanderWise

  1. Better and more focused audio descriptions of points of interest.
  2. More source data given from various different sites rather than just Wikipedia.
  3. More personalized experience by filtering data and location markers that peak the user's interest.

Built With

Share this project:

Updates