Hermes

The application used in UCLA
Used in France
Out and about filming our demo

About the project:

Inspiration

Have you ever visited a new city and been completely confused on what to do? Traveling can be incredibly confusing, especially for younger travelers or those unfamiliar with the local language. And to top that off, without an expensive or less-than-practical tour guide, learning about the culture and history of cities is challenging. As somewhat frequent travelers, our team has experienced this first-hand, often left sitting in beautiful cities scratching our heads. We came up with the goal of fixing this problem by providing travelers the historical and cultural knowledge of a tour guide while keeping the independence, flexibility, and accessibility of exploring cities on one's own.

What it does

Our app, Hermes, is an AI-powered tour guide that helps you explore the world wherever you want, whenever you want. On your terms and your time. Users enter their current location or a location of their choice anywhere around the world. Then,

Hermes displays interactive directions to five relevant attractions around the user
Hermes uses context from Wikipedia and the Gemini 1.5 API to generate cohesive stories about the history and culture of the route and destinations, with a particular focus on telling the stories of marginalized communities
Users can play audio of tour information for each destination using ElevenLabs and navigate between each location

How We Built It

We built Hermes as a full-stack app using Next.js and Flask. With the Gemini 1.5 API, Google Maps, and ElevenLabs, we were able to create a fully-featured tour guide that works for anywhere in the world.

Google’s newest Gemini model was the basis of our application and helped us present information about destinations worldwide. We made use of the 1 million token context window by scraping the web for relevant Wikipedia articles. These articles were often very long and the 1 million token context window allowed us to pass lots of information that would have been too long for the context window of other models to be the basis of the generated tour. The articles were used to generate relevant facts about destinations for the user. Gemini also allowed us to make our content accessible. We added a “kid mode” feature that makes the descriptions fun and simple for travelers of all ages and had Gemini change the response based on the feature. We also added a translate feature to our app. Hermes can now be used in over 40 different languages, allowing people all over the world to connect with the culture and history of cities using the app.

The Google Maps Platform was another incredibly important piece of our app. Our Google Maps integration enabled us to search for tourist attractions relevant to our mission of helping people connect with the history and culture of the world around them and present the five most relevant. We also used the API to display directions from a user’s location to each landmark, helping them navigate easily between attractions.

In addition, we used ElevenLabs to transcribe the transcriptions so users can listen to the information about each location as they walk between them. Compared to the native speech-to-text included within the browser, ElevenLabs provided more animate narration, with a David Attenborough-esque voice to bring Hermes additional flair.

Challenges we ran into

Bringing all of these powerful tools together proved to be quite the challenge. Our ElevenLabs integration was particularly difficult. Keeping our API key strictly server-side required a complicated series of steps. We ended up having the client send a request to our web server, and then using the ElevenLabs SDK to generate the audio. Finally, it would base64 encode the audio and send it back to the client where it could finally be decoded and played.

Our main non-technical challenge was preventing feature creep. Our app had so many components and integrations that we had to be strict about every additional feature. To make sure we were on the right track, we reached out to mentors through the Hackathons 101 and Demystifying Product workshops. Consulting with mentors helped us be more critical of the features we chose to build, preventing feature creep. By the end, we focused on providing an excellent experience with the crucial features Hermes needed to help users tour the world in a fun, simple, and culturally-rich manner.

Accomplishments that we're proud of

This was the first big hackathon for two of our members, but we still managed to create an app we’re very proud of
We were able to integrate a diverse set of technologies in a simple, cohesive way
Through sheer willpower and caffeine we stayed up for 24 hours straight to finish up the project ;)
Dinosaur logo :D

What we learned

We got to know the capabilities of LLMs and how to improve their performance and mapping. We also learned how to integrate a wide variety of services efficiently and how to utilize Railway for hosting our Flask server. And most importantly, we learned how to effectively collaborate as a team and have lots of fun in the process :)

What's next for Hermes

Hermes has a really solid core set of features, and we think it is well on its way to helping travelers everywhere explore cities and engage with their histories and culture.

Our next step is to enable Hermes to be even more flexible and customizable for users. Currently, we're using the Google Maps API to generate places. In the future, we plan on making this more customizable by enabling users to select interests (i.e. museums, ADA friendly locations) that we would then use Gemini to select destinations based on using online review sites like TripAdvisor. This opens up some cool opportunities for personalization. For example, with kids mode enabled, not only would the descriptions change, but attractions might be customized to include playgrounds or a theme park. Both bar-hopppers and barristers could get a journey that fit their personalities and tastes.

We also need to add authentication and clean up a lot of code before we release Hermes to the world :)

All in all, we are very pleased with Hermes and plan on continuing to improve the app so it can help travelers explore the beautiful history and culture of the planet we call home wherever they want and whenever they want. Happy travels!