.NET Avatar Agent SDK

The project is a .NET proof of concept for an SDK to build moving, talking Avatar Agents delivered over WebRTC. Very simplisitcally it's comparable to OpenAI's Realtime but with visual avatars.

Comment

Inspiration

Max Headroom & HAL 9000.

What it does

It aims to provide a way to include moving, talking Avatar Agents backed, by speech recognition and inference, in .NET applications. The SDK is underpinned by a WebRTC realtime communications transport to give the Avatar Agents near realtime responsiveness.

AI speak: It gives a visual embodiment to inference models. Human Translation: It allows developers to buld digital characters.

How we built it

Started from a WebRTC, real-time interactivity, UX to explore the interaction inference models. Once it was boot strapped and cloud deployed the agent was able to "evolve" by improving itself in response to feedback from its owner.

Challenges we ran into

Graphical rendering to allow the avatar to "speak" was the main challenge. For a cloud hosted avatar GPU is too expensive so the rendering was limited to CPU. Getting the speech-to-text -> inference -> text-to-speech -> avatar lip synchronisation fast enough was a challenge.

Accomplishments that we're proud of

By and large it works! Something that was not assumed at the outset.

The most amazing thing was experiencing Avatars designed by different communities (Live2D, VRoid) in the Godot gaming engine and seeing them come to "life".

Another cool hing was the ability to "power" Avatars with local ML models via the ONNX runtime for STT, TTS and inference.

Plugging in commerical inference models, such as OpenAI, was also relatively seamless but was very pleasing for the Avatar responsiveness. OpenAI's Realtime Speech-to-Speech offering made a big difference to the experience.

What we learned

Lip synchronisation was as hard as expected. Without being able to use coding agents it probably would have been a months' long task for a non-graphics/gamer developer like me. The other big learning is that the pieces fit together surprisingly well and products like this are likely to come to market sooner rather than later.

What's next for the Avatar Agent SDK

I want to do more prototyping around the Avatar motions. A lot of the Live2D models in particular come with actions like waving wand, bowing etc. which add more engagement to using the Avatar. The question to be answered is will visual avatars backed by intelligence be beneficial in line of biz apps or are they a novelty?

I also want to explore getting the Avatar to control it's environment. In the demo Max has a dynamic background of 3 sides of a cube. Because the whole video stream is being driven by frames generated in-process the Avatar can theoretically be given instructions to change itself.

Share this project:

Updates