Copoly

Real-world exploration by AI avatars, fueled by Google Maps.

Comment

Inspiration

Copoly is inspired by platforms like character.ai and some sci-fi ideas about AI and a virtual existence. At the crux of it lies the question - can we create virtual beings that can autonomously explore a place and learn about it like humans?

Humans experience places in very different ways. There is a depth of knowledge and experience about being "local" to a place. It is developed over time with layers of experience and knowledge. This project is an attempt at exploring if we can use AI agents to emanate knowledge from information.

It allows AI agents to roam free in a virtual model of the real world and see how well it can learn about places. Anyone can then interact with them to discover and learn about the places - providing a unique way to experience life at different places.

What it does

Copoly lets you create "Polys" virtual beings with their own personalities, circumstances, goals, and preferences. These Polys live in a photorealistic virtual world powered by real-time, real-world data. They explore their environment, make decisions and grow over time by having experiences, forming opinions and developing preferences for places and activities.

Viewers can interact with Polys to ask questions, get recommendations or explore locations through their experiences. The interaction allows us to learn and explore new places, while seeing how gen AI interprets and adapts to real-world data and human interaction.

Key Ideas:

  • Autonomous Polys: Polys make independent decisions and grow over time.
  • Real-World Integration: The virtual world uses live, real-world data for realism.
  • Interactive Exploration: Users can ask Polys questions and get recommendations.
  • Immersive Discovery: Explore places through the experiences and actions of Polys.

Links to some Polys - go check them out

Polys

Outdoor Champ

Social Butterfly

How we built it

Copoly is a web app built using multiple technologies.

  • It uses the data provided by Google Maps platform as the source of information.
  • It uses OpenAI for text-completions and contextual decision making.
  • It uses Convex for the infrastructure and scheduling the automations.
  • The Photorealistic map gives the backdrop for human-AI interaction supported by a UI developed with Tailwind, React, R3F and drei.
  • Realtime data is sourced from multiple sources to enrich the information available to the AI agents. Beyond this multiple other sources are used for 3D models, character development, animations and graphics.

Google Maps Platform

It integrates real-time data using Google Maps SDKs and APIs like Geocode, Places, Directions, Timezone and Photorealistic Maps to power its virtual world.

Challenges we ran into

Getting the AI model to work was a process of trial and error. The responses were difficult to control, but with prompt engineering and using the structured responses feature, I was able to achieve reliable and satisfactory results.

Another challenge was working with the 3D avatars for the Polys. It took time to get the models to look and behave correctly. Ensuring the avatars' actions aligned seamlessly with the virtual environment was a significant learning curve.

What's next for Copoly

The next step for Copoly would be to focus on enhancing engagement and interactivity. Plans include enabling more dynamic and immersive ways for users to interact with Polys and their world.

A key goal is to explore integrating the platform with Twitch, leveraging its live streaming and chat features. This would allow audiences to influence Polys in real-time, creating a collaborative and interactive experience.

Share this project:

Updates