-
-
Landing page where the user can take a video of the surrounding or upload an image from the gallery.
-
Details of the place searched along with a visually appealing video showcasing the history of the monument.
-
A map showcasing the nearby sights along with their information such as map coordinates, details and history.
-
A scrapbook of all the places searched or visited along with the uploaded video or pictures acting as a digital memory.
-
Links and references to the places visited in case the user wants to learn more later.
Inspiration
The inspiration for this project didn't strike in a lab, but amidst the silent, crumbling stones of ancient ruins. Standing before weathered inscriptions and towering monuments, I felt a profound disconnect. I didn't know the language, and the static information boards while helpful couldn't answer my specific "why" or "how."
I realized I didn't just need a translator; I needed an expert guide. I envisioned a companion grounded in historical facts that could not only speak to me but immerse me in the past through generative visuals, allowing me to compare the ruins in front of me with their former glory in real time.
What it does
Tour Buddy is an AI-powered multimodal travel assistant that uses a "Research Agent" to instantly identify historical sites or inscriptions from photos and videos. It then translates these visual insights into immersive cinematic recreations using the Veo API, allowing travelers to visualize exactly what a ruin looked like in its prime. By combining real-time grounding with high-fidelity video, it replaces static information boards with a dynamic, conversational guide that brings history to life.
How we built it
The architecture of the Tour Buddy app was built on a foundation of collaboration and cutting-edge tooling. Leveraging resources from the Gemini and Veo teams, I structured the project around two core agents:
The Research Agent: A backend powerhouse (built with Python) that processes live video feeds and queries historical databases.
The Speech Agent: An expressive frontend layer that delivers insights with a human-like cadence.
I utilized YouTube tutorials and documentation to bridge the gap between high-level concepts and working code, ensuring the frontend (HTML/CSS) could handle the heavy lifting of the backend processing.
Challenges we ran into
The road wasn't without its hurdles. The primary challenge was mapping the capacity and limits of these powerful models.
Contextual Guardrails: I had to ensure that the AI didn't "hallucinate" historical facts. Every insight generated had to be strictly grounded in the provided data.
Credit Optimization: Power comes at a price. Managing API credits became a logic puzzle in itself. I had to optimize the frequency of calls and the length of tokens to ensure that the experience was high-quality without being prohibitively expensive.
Accomplishments that we're proud of
We successfully implemented a complex multimodal pipeline that seamlessly chains a "Research Agent" for deep contextual analysis with the Veo API for high-fidelity video generation. This architecture allows the app to move beyond static data, turning raw visual input into an immersive historical recreation in real time. We are particularly proud of achieving a low-latency, "agentic" flow that provides travelers with grounded, cinematic insights that were previously only possible with a human guide.
What we learned
Building this project was a masterclass in the rapid evolution of Large Language Models (LLMs). I moved beyond the concept of "chatbots" to explore Multimodal Intelligence.
Beyond Text: I discovered how LLMs now "see" and "hear," allowing for a seamless blend of visual input and audio output.
What's next for Tour Buddy
The next phase for Tour Buddy involves integrating spatial computing and augmented reality to overlay historical recreations directly onto a user's live camera view, creating a "time-window" effect. Additionally, moving the experience to wearable devices will enable hands-free, bidirectional conversations that allow travelers to interact with history through natural dialogue. Finally, grounding these insights in verified academic databases will ensure that every cinematic recreation is as historically accurate as it is visually immersive.
Log in or sign up for Devpost to join the conversation.