-
-
Query and display of a requested Youtube video
-
Generated map data with markers
-
Optimized path using shortest distances / time
-
Generated location cards
-
Location card redirects
Wayfarer
noun. a person who travels on foot.
In recent years, travel and adventure videos on YouTube have become increasingly popular, inspiring viewers to explore the world and visit new places. However, planning a trip to a new location can be challenging, especially when trying to figure out the best route to take. By leveraging the ability to recreate adventures from media, we hope that Wayfarer embodies the spirit of wayfaring - the freedom to wander and travel as we wish.
What is Wayfarer?
Wayfarer is a web application that utilizes complex natural language processing techniques, such as Named Entity Recognition (NER) and Large Language Models (LLMs), to extract city data and geo coordinates from Youtube video transcripts to create custom travel itineraries for users. With Wayfarer, travelers can easily plan and map their adventures, creating optimal paths and allowing users the chance to relive incredible moments captured on Youtube.
A generation of markers and location cards for Iceland
A specification of features include plotting markers for locations, optimal pathing between markers, and Google Maps and Reviews for each landmark extraction.
How Wayfarer is Built
Wayfarer utilizes a clever combination of multiple natural language processing to produce accurate, natural paths taken or recommended in Youtube travel videos, even providing an ample amount of support for multiple languages. At its core, Wayfarer is an advanced Named Entity Recognition software that uses a plethora of machine learning packages to extract information from Youtube.
On a more technical level, our app is built on Flask and React servers that communicate with various APIs, first parsing the transcript of Youtube videos, formatting locations visited using OpenAI’s Davinci-003 model, and then using Spacy and NLTK’s (the Natural Language Toolkit) Punkt, Averaged Perceptron Tagger, MaxEnt NE Chunker, and the Words corpi in order to location tag and performed Named Entity Recognition to extract specific landmarks and cities.
Because cities and landmarks mentioned in Youtube videos are often casually talked about and are on their own, we reasoned that using OpenAI’s Davinci-003 model on the basis of LLM transformers’ capabilities to contextualize would give us accurate predictions through matching landmarks with the cities mentioned in the video. Additionally, the multilingual capabilities of Davinci-003 (although somewhat limited) gives it comprehension of certain locations.
For instance, if we were to have the following landmarks and cities:
Obviously, none of the landmarks would match to Paris due to the context that Davinci-003 is trained on. After pairing these cities and landmarks, we used the Geopy API to confirm the locations through their geocoding service Nominatim and extract geological coordinates of each landmark.
These landmarks were then subsequently passed to the MapBox API to produce markers for a graphical interface. Using MapBox’s optimizers, we then prepared optimal paths between these landmarks.
Challenges We Ran Into
We ran into slew of issues and bugs as we built Wayfarer. One of the largest was the methodology of extracting our named entities (cities and landmarks) - we couldn’t simply find all named entities through a standard NER package, as Youtube influencers may talk about certain places in passing (such as a comparison), but didn’t physically visit or recommend visiting those areas. Using OpenAI as a workaround allowed an intelligent approach towards extracting these named entities using contextualization.
Furthermore, we had many issues connecting the backend to frontend using Flask - namely those concerning Flask’s Cross Origin Resource Sharing feature to work with the different servers that Flask, React, and the APIs were running on. We also had major issues with being able to generate visualizations of locations described - first, an attempt to use DALL-E, and then various outdated Google Search APIs. As a result, we had to scrap a few features that we spent hours working on. Furthermore, we had an incredulous amount of bugs while running many asynchronous processes while building our app - so much so, that we started referring to the 'Searching...' loading text as a search for bugs.
Regardless, we continued to push forward and squash as many bugs as we could. Since the majority of our members have limited experience in dealing with React, are fairly new to hackathons, and web development in general, there was so that we were constantly struggling and learning from.
Accomplishments We're Proud Of
We’re incredibly proud of the fact that we were able to successfully launch and build a(n almost) full stack application with the core capabilities and features that we originally envisioned in such a short amount of time - as well as our usage of AI/ML to accurately extract information from Youtube videos.
Furthermore, we were very surprised how well we were able to work together, as everyone at almost every point in time was working on something different, yet in conjunction with the group’s progress as a whole. What’s more - as none of us were particularly experienced in designing websites, we ended up with an end result that was much more visually appealing than expected.
What We Learned
We utilized an incredible amount of techniques and technology that we were unfamiliar with to create Wayfarer, and subsequently learned so much from it. Although a sizable portion of our features were cut, we learned to utilize so many different Python packages in conjunction - pandas, nltk, openai, cohere, locationtagger, spacy, urllib, geopy - as well as how to operate as an optimized team.
Our application was built using HTML, CSS, and Javascript with React as our frontend framework, tied together with Flask. We also relied on the MapBox GL in order to generate and optimize the paths we generated in our application.
Furthermore, the large amount of technologies and techniques used in our web application forced us to continuously learn. The process was overall difficult and frustrating at times, we still had fun facing the challenges thrown at us, and were unafraid to work for a few extra hours to push for a feature we wanted to keep, no matter how hopeless it may have been.
What's next for Wayfarer
Moving forward, we hope to be able to generalize Wayfarer and offer more opportunities for LLM-assisted NER applications, as well as introduce more of the capabilities that we were unable to due to time constraints. For instance: fetching from APIs to calculate costs of travel generated from videos, computer vision and OCR to verify and better estimate more accurate geological coordinates, and sorting/generating notes from the video commentary for markers of landmarks.
An example of how object recognition could be used to validate landmarks
Furthermore, our style of LLM-assisted NER can be extended to other use-cases, such as extracting cooking materials from cooking videos and performing various calculations. Our application can also be easily modified to accompany general texts, such as extracting artifacts from guides.
Log in or sign up for Devpost to join the conversation.