Nib – An AI Companion That Evolves with Twitch Chat

Nib is an AI-powered Twitch companion that helps streamers sense chat's mood at a glance and gives viewers a little friend that reacts, chats, and evolves with the conversation.

Comment

Inspiration

During an intense game, a streamer can't read every chat message. Meanwhile, viewers want to feel that their comments are part of the experience.

I wanted to connect those two needs through a small character living in the corner of the stream.

For streamers, Nib would be a companion that makes chat's atmosphere visible at a glance. For viewers, Nib would be a familiar little friend—something they could interact with, influence, and watch evolve.

That became the central idea: stay in the game, and stay connected to chat.

What it does

Nib is an original pixel-art companion displayed through a transparent OBS Browser Source.

  • Makes chat's mood visible. Each minute, AI analyzes a batch of accepted comments. With at least five messages from one or more viewers, the results update five emotion bars and select Nib's emotional form.
  • Evolves beyond color changes. Base Nib can become Calm, Laugh, Hype, Angry, or Confused, with distinct silhouettes, accessories, and movements. A roughly 3.2-second transformation makes the change easy to notice. Quiet minutes return Nib to its base form.
  • Responds to participation. Comments trigger local actions such as jumping, chasing, eating, and sleeping. These reactions do not wait for an AI request.
  • Keeps the conversation going. Selected comments receive contextual replies through on-stream speech bubbles. Nib can also ask questions, collect votes, and respond to the results.
  • Shares situational jokes. An optional ten-minute feature combines recent chat with a delayed Twitch stream thumbnail to generate a joke about the shared moment.

The emotion display is an AI interpretation of participating chat, giving the streamer a cue rather than telling them how to run their stream.

How we built it

I built Nib with OpenAI Codex through an iterative process of describing the experience, testing it, and refining what did not feel right.

Early versions explored multiple characters. Through feedback and live testing, I narrowed the concept to one recognizable companion whose body and behavior evolve with chat. I used Codex to implement those changes, improve the transformation sequence, investigate bugs, and add regression coverage.

The application uses Node.js and Express, a receive-only Twitch IRC WebSocket connection, and Server-Sent Events to deliver updates to the browser. The dashboard and OBS overlay share one HTML Canvas renderer, with original pixel artwork drawn in JavaScript.

Immediate reactions run locally. Five emotional states and four interaction intents produce 20 distinct reactions, allowing Nib to behave differently depending on its mood without waiting for an AI response.

OpenAI handles minute-based sentiment analysis and selected contextual interactions. AI responses are checked against structured schemas, while the application owns vote counts, queues, settings, and usage accounting.

Docker provides a reproducible runtime. Playwright browser tests, unit and HTTP tests, and GitHub Actions support verification.

Challenges we ran into

Making evolution feel meaningful. Changing colors and facial expressions was not enough. I redesigned the forms around different silhouettes and added anticipation, charging, a silhouette transition, a reveal, and a landing.

Keeping Nib visible and responsive. Early rendering failures could make the character disappear after time gaps or bursts of input. Working with Codex, I revised sprite drawing and recovery behavior and added regression tests for these cases.

Handling unreliable connections and asynchronous AI. A Twitch connection could appear connected while no longer receiving messages. Heartbeat checks, reconnection handling, and connection identity guards improved recovery. Separate AI windows and stale-result checks prevent delayed responses from overwriting newer state.

Balancing personality with attention. Nib needed to feel alive while leaving gameplay readable. This shaped the corner placement, compact movements, reply pacing, and separation between instant actions and slower AI interactions.

Accomplishments that we're proud of

I am proud of turning one character into a shared interface for two audiences: a visual cue for streamers and a playful participant for viewers.

The project includes six original forms, expressive transformations, contextual conversation, polls, and image-informed jokes.

Verification includes 184 passing unit and HTTP tests on Windows and Docker Linux, six browser test suites validated on Windows, and real Twitch tests covering reactions, replies, two-account participation, and emotional evolution.

The demo also shows the actual Codex development process alongside the working Twitch experience.

What we learned

I learned that a character's appeal depends on timing, anticipation, and continuity as much as its artwork.

I also learned that AI works best here when it has a specific role. Fast interactions benefit from predictable local logic, while interpretation and conversation benefit from generation.

Codex was most useful when I supplied concrete feedback, reproduced failures, and checked the resulting changes against the real stream.

What's next

Next, I want to test Nib in longer sessions with more streamers and communities, refine multilingual replies and humor, and improve onboarding and customization.

I also want to study whether streamers find the visual cues useful and whether viewers enjoy returning to the same evolving companion.

The goal is to keep Nib small enough to belong in the corner of a stream, and expressive enough to become part of its community.

Built With

Share this project:

Updates

Submission history