World Interact

WorldInteract pushes the boundaries of multimodal interaction beyond images, audio, and text. Thanks to the power of Gemini, it transforms user input into immersive, automated journeys.

Comment

Inspiration

From age 10 to 24, I've witnessed Google Earth's remarkable evolution in terrain detail and visual quality. Today, I regularly explore the 3D tilesets of various locations on Google Maps web, noting that Chile, my home country, currently has only three cities covered for now...

Last year, the integration of these tilesets into game engines like Unreal and Unity captivated me. I dedicated the year to experimentation, testing vehicles, incorporating BIM models, implementing greedy flight algorithms with raycasters, and exploring other simulations. I firmly believe that the next leap in multimodal AI lies in the generation and refinement of photogrammetry through 3D inpainting, ultimately placing the world within our grasp.

This proposal aims to offer:

  • Sustainability: Enhance users' understanding of environmental resources to promote their responsible utilization.

  • Real Estate Investment: Provide panoramic views of properties with contextualized information about the neighborhoods.

  • Culture and History: Offer tours of historical sites significant to humanity.

  • Tourism and Travel exploration: Promote enriching tourism experiences.

Finally, a significant inspiration stems from the multi-platform virtual tours that were pioneered in the early 2000s.

What it does

Imagine saying, "Show me the best castles in Germany," and watching as Gemini intelligently plots a virtual tour, complete with visuals and audionarration.

From historical explorations like "Show me the expansion of the Roman Empire" to practical applications like "Show me a house on 253 Smith Av ,San Jose, CA," or fiction like "Show me the locations used in the kingdoms of your favorite movies / tv shows, "Geographic Faults or Climatic Issues".

How?

Unlike simply using Gemini for coordinates and places, or utilizing Geocode for place coordinates, or Places Text Search API, I have created an fine-tuned Decider Agent that can decide, based on any input, which of these three modes to use. This approach helps save resources, as the Gemini API is less expensive than Places API, and it is used only when necessary.

flowchart TD
A[Input User] -- text --> B{Decider Agent};
B --> C(Mode 1: Trust in Gemini);
B --> D(Mode 2: Hybrid );
B --> E(Mode 3: Specific Places);
Mode 1 Mode 2 Mode 3
places names Gemini Gemini Gemini
places coords Gemini Places Geocoding Places NearbySearch
advantajes less expensive middle most accurate
disadvantajes less accurate middle most expensive

With the help of Gemini, the decisor agent converts the text to key parameters like zone_name, isMacroZone, num_routes, extraContext ,etc

The modes:

  • Gemini-Driven Location and Coordinates: Ideal for macro-zones or well-known places such as historical sites".
  • Gemini + Geocoding: For most cases, it is the ideal combo using the Geocoding API.
  • Google Places Type Classification: This mode receives the parameters granted by the Decider Agent and compares the trained parameter 'placeContext' with the placeTypes saved in a file in the backend , the same as PlacesTypes API. It also considers the 'zone_name' and returns that to perform reverse Geocoding. Then, it performs a nearbySearch with the 'placeContext' labels and given 'num_routes', generating routes or a single destination if that's the case.

Finally, Gemini generates a narration with audio narrated by text-to-speech, and places card for mode 3.

How I built it

Node js backend, Frontend vanilla, use Google Maps Place, Google Text to Speech, and Gemini of course.

  • Node js for Backend.
  • Frontend vanilla.
  • Google maps Places API ,Geocoding (Inverse too), Place Types, Markers and othes.
  • Google Text-to-speech AI.
  • Gemini API.
  • Sound Effects no copyright by Youtube Studio.

Challenges I ran into

I will list the main challenges:

  • Gemini API version updates: With the release of new versions, even if I used the same one as before, the results changed, affecting performance and error handling. In fact, it delayed my delivery because with the release of Gemini Experimental 1114, I had to redo the prompt engineering, also bugs with JSON output.
  • Geocoding API has some failures in English with simple requests.
  • Animation synchronization: I saw the new flyTo tools, but too late because I had already made my own and it made me spend a lot of time getting it to match the audio and text description when using the Stop button.
  • Fine-tuned : I had problems integrating the fine-tuning model with the REST API, so I decided to use a list of inputs and outputs as parts, and it yielded good results with Gemini 1.5 Flash, flash--002 and pro.
  • Live events and Gemini grounding feature: Working with JavaScript, I noticed that the Gemini API grounding documentation appears to be available only in Python at this time."

Accomplishments that I am proud of

My mode 3 is comparable to the new Places TextSearch feature, but provides a richer understanding due to its ability to accept inputs like: Where should we go for a summer holiday in Adelaide with three 10-year-old children? I'm looking for a condo in a middle-class Los Angeles neighborhood around $1 million.

It can also be seen as the automation of using the Aerial View API. Its application is already visible on social media, particularly for real estate and historical content creation.

I expect numerous projects focused on navigation and storytelling using place names, Gemini, and geocoding as inputs. However, I'm particularly proud of my value proposition, which includes the integration of a resource-optimizing decision agent.

What I learned

I learned a great deal about practically all the resources the Google Maps Platform can offer, including the associated costs. This will significantly help me professionally and in future projects.

What's next for World Interact!

Significant code-level improvements have been made, such as migrating the frontend to Flutter or React, restructuring, and utilizing the new FlyTo functionality. The following functionalities particularly stand out to me:

  • Specialized fine-tuning subAgents for: real estate, sustainability, and history.
  • Live events: The new Gemini grounding feature will allow us to showcase locations of automated live events. Real-time news and daily event updates are the next step.
  • Integration of BIM/AECO models (.IFC) glTF format.
  • Chat mode with live voice integration and spatial context, for example, "turn left," "move forward 10 km",
  • Enabling user storage and deployment.
  • Integrating Google AI Vision or Vertex for screen recognition.

Infinitely scalable, there's still a lot to do!

Share this project:

Updates