Identity Matrix

.

Comment

Inspiration

What if your “online presence” didn’t stop when you closed your laptop?

Identity Matrix is a real-time multiplayer simulation where your avatar transforms into a stateful agent. When you log off, your AI keeps running: it remembers how you talk, what you care about, who you avoid, and who you keep coming back to.

Step 1: upload your identity to the Matrix. Step 2: find out you...

  • joined a study group,
  • got into an argument with someone’s Identity,
  • and somehow became the unofficial mayor of the library.

What it does

We developed a real-time multiplayer virtual world with four core features:

  • A realtime multiplayer world shared with other players (and their AI stand-ins) in real time. This world was entirely hand-made - the background and the layout was entirely made within the hackathon!
  • World and agents that never sleep. After you log off, your agent continues to exist with your personality, interests, and communication style. It navigates the world, manages basic needs (hunger/social, based on the tailored metrics we designed), and proactively seeks interactions.
  • Spectator mode to observe the ecosystem. You're able to watch players and agents form conversations, avoidance patterns, friendships, and enemies.

Our proudest feature is the Continual Learning loop we developed. The model iteratively learns from your mannerisms and interactions, storing this into our custom memory bank, optimized to remember both important things and minor details. Your alter Identities respond in your communication style using your memories and social context from the world, mimicking you!

(We sometimes get confused if we're talking to another human, or their Identity..)

How we built it

Architecture

Component Stack What it does
web/ React + TypeScript + Vite + TailwindCSS + Phaser 3 Game client and frontend. Phaser renders the world, while React handles UI/overlays
realtime-server/ Node.js + TypeScript + WebSockets Server-authoritative simulation with a 10Hz tick, entity sync, AI coordination loop
api/ Python + FastAPI Agent controller, meaning onboarding, conversation generation, decision scoring
world/ Pure TypeScript Deterministic game logic, collision, pathfinding, conversation state machine
supabase/ PostgreSQL + Auth + Storage Persistent users, positions, memories, personality profiles, sprite storage

The AI stack

  • Grok 4 (via OpenRouter)

    • onboarding interview -> personality seed that initializes the AI's impression of your memories
    • conversation responses
    • lightweight sentiment/intent signals for social memory updates

  • Gemini 3 Pro Image

    • sprite generation from photos (pixel-art style), including animations and a sprite table!
    • was crucial to personalizing the characters and making it feel like your real alter Identity

  • Utility-based decision engine (Python)

    • a compact policy: score actions (wander / approach / talk / avoid / idle)
    • softmax action selection to keep behavior non-robotic but still explainable

Key technical decisions

  1. Server-authoritative simulation
    Movement is validated server-side. Clients can’t teleport, desync, or cheat their way through collision.

  2. Deterministic world core
    The world/ engine is pure TypeScript and deterministic so the simulation stays consistent across clients and reconnects.

  3. LLM isolation boundary (sandboxed)
    LLMs can generate dialogue only — they never directly control movement or world state.
    This prevents “hallucination exploits” (e.g., the model deciding it teleported or changed the map).

  4. On-demand agent decisions
    Agents request their next action only when idle (event-driven), instead of burning tokens on a fixed timer.

  5. Reservation-based movement
    Multiple entities moving simultaneously resolve conflicts via a reservation table, preventing tile collisions and deadlocks under concurrency.

Challenges we ran into

Agents are hard. Very hard.

  • Getting agents to not slam into walls required real pathfinding + collision constraints
  • Two agents stepping into the same tile at the same time turned into a full-on concurrency problem
  • How do you make your agent want to do something???? (we found a solution to this by sapping their energy and increasing their curiosity :))

Game dev is also hard (especially at 3 AM).

  • Phaser is powerful, but the learning curve is vertical (we literally learned this library today...)
  • Syncing React state with Phaser state without creating a desync monster
  • WebSocket reconnection logic that doesn’t duplicate entities or resurrect ghosts (too many times did we have to reset our Supabase)

Sprite generation: non-determinism is pain.

  • Image models love improvisation
  • We needed strict prompting and verification
  • Direction detection required pixel-level analysis (our direction detection algorithm is 200+ lines)

Accomplishments we’re proud of

  • It actually works. Realtime multiplayer + persistent agents + conversation.
  • Agents develop social preferences: they avoid people who were rude and seek out people they like.
  • You can log off and your AI continues living :D

What we learned

  • Utility-based agents are shockingly elegant once the scoring is right
  • WebSockets are great until reconnection edge cases show up with a bat
  • Phaser + React can coexist, but they are not exactly best friends
  • “Consistent personality” is 80% prompt engineering + memory hygiene
  • Image generation needs extremely specific constraints to behave deterministically
  • Game dev is basically state machines all the way down

What’s next for Identity Matrix

Immediate improvements

  • More locations (shops, parks, libraries)
  • Mobile support

Long-term goals

  • Long-term memory + goal formation (your AI develops life goals over time)
  • Emotional contagion (mood spreads through proximity/social graphs)
  • Day/night cycles affecting behavior
  • Emergent gatherings and events
  • Voice chat with your AI avatar (voice cloning maybe????)

More than anything, we'd love to simulate this over a long time period, with a large userbase! We built this project to satisfy our curiousity about what how our identities could change over time, within parallel universes. Would they change? Would we stay the same? Identity Matrix allows us to test and find out.

We hope you enjoyed learning about our project as much as we enjoyed making it!

- Alvina, Ethan, William and Zac

Built With

Share this project:

Updates