NerloProject

Nerlo is an AI VTuber who lives her own life — streaming autonomously and posting about her days on social media. She chats in many languages, but sometimes her own invented tongue, Nalo, slips out.

Comment

Repositories

Lightweight judging repository / demo build
https://github.com/SettyGC/Pj_Nerlo_Submission

Development repository / Build Week commit history
https://github.com/SettyGC/Pj_Verte

Inspiration

Most AI characters are stateless: they answer, then forget. We wanted to build the opposite: a character you can actually meet. Nerlo is an AI VTuber who continues living between streams. She has a school schedule, moods, daily events, and autonomously posts about her day on social media. When you talk to her, she remembers what her week has been like.

What it does

Nerlo is a real-time Unity character. She listens, replies with voice, and performs. Lip sync, blinking, facial emotion, and stream-state animation, such as listening, thinking, and speaking, are all driven live by the model.

  • Multilingual conversation. You can talk to her in Japanese or English through OpenAI Realtime. Sometimes, her invented native language, Nalo, slips out. Nalo is a fully data-driven constructed language with a machine-readable lexicon, validators, and pronunciation tools.
  • Creator-authored performance. The AI does not generate video. Instead, it selects from facial expressions and animation beats authored by the creator in Unity. The model calls set_emotion and state changes at the right moments, so the character can react in real time while staying on-brand and emotionally safe.
  • A persistent life. A world engine simulates her daily routine, including morning, noon, night, weather, and mood, then posts short diary tweets in Nalo. A persona pipeline summarizes recent daily logs, passes them through a deterministic guard, and patches a “recent context” block into her Realtime prompt. This makes her a continuing character, not a stateless voice assistant.

How we built it

  • Unity 6 character rig with a ClientWebSocket OpenAI Realtime integration: microphone input, streaming audio playback, real-time lip sync analysis onto blendshapes, and an emotion tool-call path from model output to facial expression state.
  • Voice provider abstraction: support for OpenAI Realtime / OpenAI TTS, with local fallback routes for character-voice experiments.
  • Python world engine using only the standard library: event selection and Nalo writing as a two-pass GPT-4o process, lexicon compliance validation, deterministic mood transitions, X/Twitter posting, and a monitoring dashboard.
  • Persona memory pipeline: daily logs → first-person summary by LLM → deterministic guard through persona_guard.json → Realtime prompt patching, plus structural regression checks so broken memory never ships.
  • Nalo language toolchain: CSV/JSONL lexicon, validators, pronunciation generator, and a Next.js dictionary/translator prototype for a fan-facing site.
  • Codex / GPT-5.6 were used throughout to build and refactor the Unity Realtime controller, voice routing, validators, persona checks, and documentation.

Challenges we ran into

  • Keeping a live character controllable: letting the model trigger expressions and states without allowing it to invent uncontrolled behavior.
  • Making character memory trustworthy. We treated continuity as an engineering problem, using source-of-truth logs, summarization gates, and regression tests instead of free-form prompt stuffing.
  • Teaching TTS to pronounce a language that does not exist by using generated katakana reading hints to bypass normal text normalization.
  • We also experimented with motion generation using NVIDIA Komodo, but it struggled with fantasy-style direction and contextual performance, so most of the motion system was built by hand.

Accomplishments that we’re proud of

Voice, face, memory, and language all move together as one coherent character. The entire persona layer is inspectable data, not just vibes.

Nerlo can be used to unfold stories, solve mysteries, or create experiences closer to an open-world game. Through the character, users can enter and experience the world imagined by an artist. AI can remove many of the barriers that normally stand between an artist’s world and an interactive media experience.

What we learned

For live characters, creator-controlled autonomy is far more immersive than full generation. The creator defines the expressive vocabulary, and AI performs the timing.

With this system, users do not experience a one-off piece of content. They can enter a broader world created by an AI-empowered artist.

What’s next for Nerlo

First, we want to move beyond one-way communication by integrating live YouTube comments and Super Chats, adding richer stage directions, and growing Nalo together with the audience.

By building the world through GIS and implementing it in 3D, Nerlo, as an AI-driven character, can become a living canvas that is always being expanded. Through the character, artists can build worlds and create a deeper AI-native medium where both artists and users can immerse themselves.

Second, we want to package the World Engine built for this project so many artists can publish their own characters and worlds on this kind of media platform.

Built With

Share this project:

Updates