Elvenlabs x Agora Agents

ElevenLabs ConvoAI SDK uses direct web-socket connections which are unstable for the complexities of end client delivery. Using Agora as the audio transport layer from client to server.

Comment

Inspiration

Web-sockets are great for server to server communication but can cause headaches when trying to deploy web-sockets at scale due to the complexities of last-mile delivery.

What it does

Our project takes the ElevenLabs ConvoAI Python SDK and implements it in a server-side application that joins an Agora channel, subscribes to the remote user, and takes their audio and pipe's it to ElevenLabs web-socket.

TLDR; Makes it easier for developers to adopt ElevenLabs Convo AI without the headaches of setting up streaming infrastructure.

How we built it

Using ElvenLabs ConvoAI Python SDK, Agora Python Server SDK, for the backend. Using Lovable to generate a client app that uses Typescript, NextJS and Agora for the frontend. Including a deployer script for pushing the project to your aws/digital-ocean infra (GCP in progress)

Challenges we ran into

Handling the audio stream, Agora streams are 24khz but ElevenLabs uses 16khz.

Accomplishments that we're proud of

Getting this working.

What we learned

How to use ElvenLabs ConvoAI Python SDK for custom audio streaming.

What's next for Elvenlabs x Agora Agents

I plan to write a blog recapping the experience and evangelizing within our respective the developer communities.

Built With

Share this project:

Updates