Tavi

A live interactive streaming companion.

Comment

Inspiration

TikTok gives you a streak pet when chatting with friends. Duolingo has a world-renowned mascot that has helped bring in more users to the platform. Taking inspiration from these giants, I decided to build "Tavi", a streamer companion that lives and breathes on your overlay but interacts with you and your viewers.

What it does

Tavi is a simple model that just augments the streaming experience for both streamer and viewer. Tavi can react to messages. When someone gifts you a sub or when another streamer raids your stream, Tavi reacts and engages both the streamer and audience.

How we built it

Tavi’s Studio and landing page are built with React and TypeScript. A local Node service handles settings and stream events, then synchronizes Tavi between Studio and the OBS Browser Source. Her 2D character uses a layered rig and reusable motion and expression systems. Twitch events come through EventSub. The built-in reactions use rules and cooldowns, and streamers can optionally connect their own ChatGPT account to direct expressions and captions.

Challenges we ran into

One of the biggest hurdles I encountered was building and refining the Tavi model to incorporate more complex movements and behaviours.

Codex is really amazing with UI, and with the right agent skills, there's pretty much no hard limits on what you can do. However, when it comes to character design, I find that Codex struggles to come up with something much more polished and tangible without a lot of hand-holding.

As a result, I took several dedicated sessions to get the proper model foundations and rigging correct before building a library of expressions on top of that foundation.

Accomplishments that we're proud of

Within these 8 days, I have gone from understanding very little about character and motion design to understanding behaviour modeling and motion graphics, and have learned how to take a more systems-based approach to building this project. As a Biochemistry PhD student, this is but a fraction of the things I am proud of in this project, and I am genuinely excited to continue building more with Codex.

What we learned

Character behavior depends on more than expressions alone: the rig, movement, timing, and interaction rules need to work together. We also learned that Codex works best for character design when guided with clear references and specific feedback, and that simulated events should be clearly distinguished from live stream activity.

What's next for Tavi

Right now, Tavi acts more as a viewer and (sorry Tavi) a jester. The next step would be to grant Tavi more control of her behaviour and her reactions based on the learned behaviour of the streamer themselves, and the kind of audience they bring in. All within foreseeable guidelines, of course.

Tavi's future development could be split into two phases:

Behaviour

Currently, Tavi's movements, idle, and interactive states capture a wide range of emotions and possible streamer-viewer interaction moods. However, I believe that by allowing for unique behaviour and expressions, perhaps learned from the streamer and their audience, Tavi can deliver a more personalized and authentic experience for both parties, for both streamers and viewers.

To do this, though, more fine-grained "expression primitives" need to be defined, and more behavioural parameterization is needed. Currently, the user can (in the studio companion app) link Tavi to their ChatGPT account to allow for agentic control of the expression. In the future, this will be used to deliver text responses; more behavioural work and studies need to be done and carried out.

Customization

At the moment, everyone gets the same default Tavi model. This works as a proof of concept and isn't too bad for a hackathon project. However, the Twitch community is so diverse that we would be missing out, and it would be unfair to not give these diverse communities a chance to have Tavi represent them and what they stand for.

So the next thing after this would be to also conduct more research into character customizability, and look for how to enable customizations that bring communities together. For example, if streamer A has a themed streaming marathon, Tavi could be customized to look, dress, or behave in line with that theme. Or perhaps it's a Halloween-themed stream? Then we could have Tavi be optionally set to behave with certain archetypes of the streamer or the chat's favorite Halloween monsters.

Built With

Share this project:

Updates

Submission history