NYC-Mon

NYC-Mon is a Tamagotchi-style AI adventure where you hatch, raise, and bond with living Mons in a futuristic NYC—powered by NVIDIA AI for real-time conversations, emotions, and evolving personalities.

Comment

Inspiration

What it does

How I built it

Challenges I ran into

Accomplishments that I'm proud of

What I learned# NYC-Mon: Your AI Companion in a Living New York City

What if your Tamagotchi could actually talk to you, remember you, develop a personality, and grow alongside you?

NYC-Mon brings that idea to life using NVIDIA AI and Nebius Token Factory. It's a next-generation virtual pet experience set in a futuristic New York City, where players hatch, raise, communicate with, and build genuine bonds with intelligent creatures called Mons.

Instead of simply tapping buttons to keep a virtual pet alive, you interact with a companion that can respond to your voice, understand your intentions, remember meaningful experiences, and take actions that affect its actual well-being.

Inspiration

NYC-Mon was inspired by the nostalgia of Tamagotchi, Pokémon, and Digimon, combined with the energy, diversity, and personality of New York City.

Traditional virtual pets are charming, but their behavior is usually predictable. Feed them, play with them, watch their meters change, and repeat.

I wanted to explore something different:

What happens when a virtual pet becomes an intelligent companion instead of a collection of predefined animations and responses?

What if it recognizes how you care for it? What if conversations feel different as it grows? What if its personality, voice, and relationship with its owner evolve over time?

That question became the foundation of NYC-Mon.

The larger NYC-Mon universe imagines New York decades into the future, populated by extraordinary creatures inspired by the city's wildlife, neighborhoods, and cultures. For this hackathon, I focused on bringing the most personal part of that universe to life: the relationship between a player and their Mon.

What It Does

NYC-Mon combines virtual pet gameplay with a persistent AI companion.

Players can:

  • Hatch and raise a Mon: Begin with an egg and care for a growing digital creature.
  • Have AI-powered conversations: Interact with a companion powered by NVIDIA Nemotron through Nebius Token Factory.
  • Influence real game state: Ask your Mon to eat or interact with it naturally. The AI can select authenticated game actions rather than merely describing them.
  • Build a lasting relationship: Feeding, care, and interaction contribute to persistent companion state and bonding.
  • Develop personalized memories: With player consent, the companion can retain meaningful preferences across conversations, with controls to export or forget them.
  • Experience a character with its own voice: Spoken responses and growth-stage voice design help make each Mon feel distinct.

The central idea is simple: your Mon shouldn't just say it's hungry or happy. Its needs, actions, and relationship with you should matter inside the game.

How I Built It

NVIDIA Nemotron + Nebius Token Factory

At the heart of NYC-Mon is NVIDIA Nemotron-3.5 Lightning, accessed through Nebius Token Factory.

Nemotron provides the companion's conversational reasoning and tool-selection capabilities. Instead of relying on hardcoded dialogue trees, the system can interpret a player's request and determine when a game action is appropriate.

For example, when a player asks to feed their Mon:

  1. The request reaches Nemotron through Nebius.
  2. Nemotron selects the feed_mon tool.
  3. An authenticated Model Context Protocol (MCP) service validates the request and the player's ownership.
  4. The backend executes the care action and persists the updated state.
  5. The companion responds using the result of the actual game action.

In a verified integration test, the AI-selected feeding action increased a Mon's fullness from 50% to 81.1%.

That small interaction demonstrates something important: the AI isn't simply role-playing a virtual pet. It can affect the game through controlled, verifiable actions.

Voice and Emotional Expression

I integrated NVIDIA Riva/Parakeet speech-recognition components into the companion architecture and used ElevenLabs for synthesized character speech.

Voice pitch and delivery are designed to vary with a Mon's growth stage, so a young creature can sound different from its evolved form.

I also developed simulator integrations involving NVIDIA Audio2Face-3D and Audio2Emotion, exploring how speech and emotional signals could drive a creature's expressions and animation.

The hosted hackathon experience uses simpler mouth-animation behavior while the full facial-animation pipeline continues toward integration.

Persistent Memory and Gameplay

The companion's intelligence connects to a persistent game-state system rather than existing only inside an AI conversation.

With player-approved memory, a Mon can retain meaningful information across separate sessions. Authentication, ownership checks, and memory-isolation controls ensure that one player's companion data cannot be accessed by another player.

I also implemented memory export and forgetting controls because players should decide what their companions remember.

Application Architecture

NYC-Mon is built using a TypeScript monorepo with React Native, Expo, Next.js, and shared game and UI packages.

The architecture incorporates:

  • Nebius Token Factory for hosted NVIDIA model inference.
  • NVIDIA Nemotron for conversational reasoning and tool selection.
  • MCP for authenticated interactions with companion gameplay.
  • NVIDIA Riva/Parakeet for the speech-recognition pipeline.
  • ElevenLabs for spoken character responses.
  • Neon PostgreSQL for persistent data.
  • Better Auth for authentication and player identity.
  • Payload CMS for game and content management.
  • Three.js, WebGPU, Skia, and Rive within the broader rendering and animation architecture.

The project is designed around a larger cross-platform vision, allowing the AI companion to eventually extend beyond the current experience into the full NYC-Mon world.

Challenges I Faced

Making AI Actions Real and Safe

One of the biggest challenges was connecting language-model decisions to persistent gameplay without giving the model unrestricted access to the game.

A conversational response alone isn't enough. When an AI says it fed your Mon, the game must actually reflect that action.

I built authenticated MCP tools with ownership validation, structured requests, and persistent state changes to bridge that gap.

Handling Real Provider Responses

Integrating hosted inference exposed edge cases that weren't obvious from mock responses.

For example, Nebius could return tool_calls: null when no tool was selected. The initial response schema rejected that valid result.

Content-filtered responses also needed explicit handling rather than being treated as malformed output.

Fixing those issues reinforced the importance of testing against real model responses instead of relying exclusively on simulations.

Building Memory Without Sacrificing Privacy

I needed a companion capable of remembering meaningful experiences while keeping each player's information isolated.

Implementing consent-based memory, session continuity, authenticated access, export, and forgetting required treating memory as part of the product architecture—not just another prompt.

Synchronizing Speech, AI, and Animation

Making a creature feel alive requires more than generating dialogue.

Speech recognition, model inference, voice synthesis, animation, and gameplay all operate on different timelines. Coordinating them while keeping the experience responsive remains one of the most interesting technical challenges.

What I Learned

This project changed how I think about building AI-powered characters.

The most important innovation isn't that a virtual pet can talk. It's that the pet can understand a request, take an authorized action, remember the outcome, and continue building a relationship.

I learned how to integrate NVIDIA model inference through Nebius, connect model-selected tool calls to authenticated game actions, build persistent and privacy-conscious AI memory, and design voice and animation systems around living characters.

I also learned how much engineering exists between an impressive AI demonstration and an interaction that players can genuinely trust.

Accomplishments I'm Proud Of

  • Running real NVIDIA Nemotron inference through Nebius Token Factory.
  • Demonstrating an AI-selected action that changes persistent Mon gameplay state.
  • Preserving consented companion memories across fresh sessions.
  • Implementing ownership isolation, memory export, and forgetting.
  • Connecting conversational intelligence with the foundations of a larger creature-raising game.
  • Prototyping character voice and emotional-animation systems.

What's Next

The hackathon experience is the beginning of NYC-Mon.

Next, I want to complete real-device voice-interruption testing, connect the full NVIDIA Audio2Face/Audio2Emotion animation pipeline to the hosted companion, and expand the experience into deeper creature growth and evolution.

The longer-term vision includes exploring futuristic New York, discovering new Mons, developing relationships, battling, and encountering intelligent characters throughout the city.

Ultimately, I want NYC-Mon to demonstrate a new kind of interactive entertainment: AI companions that aren't limited to answering questions, but can live, learn, evolve, and participate in a world alongside the player.

Tamagotchi gave us something to care for. NYC-Mon gives us something we can grow with.

What's next for NYC-Mon

Built With

Share this project:

Updates