The Friendship Village

Inspiration

Every friend group has the chat that slowly goes quiet. Someone says "we should hang out soon," someone replies "next week?", and then nothing happens. Nobody stopped caring. Keeping a group together just takes effort, and that effort usually falls on one person until they get tired of it.
We wanted to give that job to AI. Most AI tools help you do your homework or answer your email. We wanted one that does the quiet social work instead: nudging people to check in, noticing when someone has gone silent, suggesting something to do together, and making everyone's small wins visible to the group.
We also wanted it to feel like a place, not a productivity app. Games like Animal Crossing and cozy town builders make people want to come back every day, so we built a little village where your friends actually live, look like themselves, and talk in their own voices. When the group shows up for each other, the town grows. When everyone drifts, the town falls apart until you fix it together. The goal is simple: get your people together.

What it does

The Friendship Village turns your friend group’s weekly goals into a shared 3D town in the browser. Each friend gets a house, an avatar that looks like them, and a feed of progress.

Daily check-ins: Post progress with photo evidence to earn XP and coins. Spend coins on outfits, furniture, and house upgrades.
The AI Mayor: A Gemini-powered Mayor reads the week’s activity and writes village news, nudges, and challenges. When disaster strikes, the Mayor assigns real-life repair tasks, like working out together or calling a friend. Verified photo check-ins rebuild the town.
You, but a villager: With consent, a photo becomes a walking 3D character and a talking portrait with your own (ElevenLabs) voice.
Real-time multiplayer: See friends walk around town live, visit their homes, and decorate a shared Community Hall together.
iMessage loop: Get nudges by text, reply with progress or photos, and send “Villager-grams,” short talking-avatar messages.
Stakes (optional): Friends can put devnet SOL into challenge pots with payout rules locked at creation.

How we built it

Village is a web app with a 3D town at its center. The frontend is React, TypeScript and Vite, with the town rendered in React Three Fiber (three.js). A FastAPI backend in Python owns all the game state in Postgres with TimescaleDB, and pushes live updates over a WebSocket so everyone sees each other walk, sit and decorate in real time. House interiors sync through TigerData using Postgres LISTEN/NOTIFY.
Avatars come from one photo and one voice memo. SAM 3 segments the photo and pulls out your colors, Gemini's image model turns your face into a chibi version, and Hunyuan3D builds it into a 3D head on your villager. LAM reconstructs a 3D Gaussian-splat head from the same photo, and Audio2Expression plus an ElevenLabs voice clone make it talk. The same SAM 3 → Gemini → Hunyuan3D pipeline turns photos of real objects into furniture for your house. All of the GPU work runs on Modal, called by a small worker on our server.
Gemini also plays the Mayor: it writes the weekly Gazette, picks real-life repair tasks when a disaster hits, and reviews photo evidence for check-ins. The Mayor reaches people outside the app through Photon's iMessage agent, which sends Villager-grams (your friend's avatar saying a message on video) and accepts check-ins and photos by text. Competitions can carry a Solana devnet prize pot, with an Anchor program that pays out winner-take-all, proportionally or by ranked split.
Everything runs on a Vultr VM behind Caddy at thefriendshipvillage.vodka. We built it over the weekend with a small team plus many AI coding agents working in parallel, with consent checks, daily cost caps and caching on every paid AI call.

Challenges we ran into

Real-time multiplayer was the hardest part. At first walking felt laggy, but we added client-side prediction and a short smoothing buffer. We also hit some other bugs: shared demo accounts kept kicking each other offline, one extra field in a message made the server reject every move, and villagers walking on autopilot kept bouncing back and forth in place. There were certainly a lot of bugs regarding interactions between users but most of the blaring ones have been ironed out for now.
Getting the computer vision pipeline into production took a lot of work. SAM 3, LAM, Audio2Expression and Hunyuan3D started out on a local GPU. We moved them to Modal so the site works when that PC is off. GPU cold starts took two to three minutes, which broke our video messages until we gave them more time. In the last hour, photo uploads from iPhones failed because of an unusual JPEG format. Additionally, getting villager models to be acceptable was a bit fiddly. Heads floated in front of the neck and hats sat in the wrong place until we fit each one to the head's actual shape. Outfits would clip improperly and form two torsos or none at all.
Costs and privacy also shaped a lot of decisions. For costs, every paid call (voice, LLM, GPU) has daily limits and caching and we hit a lot of our own limits during testing. For privacy, real faces and voices are opt-in, and we kept them out of the public repo. You also have to opt-in to allow others in the village to use your voice for custom lines.
A funny one was trying to get Gemini, the Mayor, to give villagers real life tasks like "call a friend" or "exercise" instead of the ones it kept trying to give us like "water the charred fence".

How we built it

Village is a web app with a 3D town at its center. The frontend is React, TypeScript and Vite, with the town rendered in React Three Fiber (three.js). A FastAPI backend in Python owns all the game state in Postgres with TimescaleDB, and pushes live updates over a WebSocket so everyone sees each other walk, sit and decorate in real time. House interiors sync through TigerData using Postgres LISTEN/NOTIFY.
Avatars come from one photo and one voice memo. SAM 3 segments the photo and pulls out your colors, Gemini's image model turns your face into a chibi version, and Hunyuan3D builds it into a 3D head on your villager. LAM reconstructs a 3D Gaussian-splat head from the same photo, and Audio2Expression plus an ElevenLabs voice clone make it talk. The same SAM 3 → Gemini → Hunyuan3D pipeline turns photos of real objects into furniture for your house. All of the GPU work runs on Modal, called by a small worker on our server.
Gemini also plays the Mayor: it writes the weekly Gazette, picks real-life repair tasks when a disaster hits, and reviews photo evidence for check-ins. The Mayor reaches people outside the app through Photon's iMessage agent, which sends Villager-grams (your friend's avatar saying a message on video) and accepts check-ins and photos by text. Competitions can carry a Solana devnet prize pot, with an Anchor program that pays out winner-take-all, proportionally or by ranked split.
Everything runs on a Vultr VM behind Caddy at thefriendshipvillage.vodka. We built it over the weekend with a small team plus many AI coding agents working in parallel, with consent checks, daily cost caps and caching on every paid AI call.

What we learned

One of the hardest product problem wasn’t 3D or ML. It was making goals and competing with friends towards those goals an enjoyable experience. The user experience, competitive incentive, and fun progression was what made the core of the app.
AI integration in applications is very useful when it suggests and both a human mediator and deterministic code makes final decisions.
Iterative design requires much coordination between developers and a clear understanding of the user facing experience, so development continuously delivers on what users want and their requirements. We consistently tested how users would experience our app before fully implementing each feature.

Accomplishments that we're proud of

Getting the full pipeline working end to end, from real-life photos to a walking, talking 3D version of you and your friend. Shipping a live, deployed multiplayer town with real-time interactibility. Designing a village people want to return to by figuring out what genuinely motivates each of us and combined our different ideas to appeal to a broader group so daily check-ins feel like a reward rather than just goal tracking.

What’s next for The Friendship Village

Fully generated animated bodies, not just heads.
Pets, backyards, and more house customization.
Turning Auth0 sign-in on by default and opening the village to more friend groups.
Deploying the v2 Solana pot program and testing real stakes on mainnet with opt-in groups.