Generative UI Toolkit For ElevenLabs Conversational AI Agent

Graphical user interfaces are powerful, but they can be tedious.Audio interfaces feel natural, yet some information is difficult to convey through voice alone. Why not combine the best of both worlds?

Comment

What it does

This package equips ElevenLab's Conversational AI agent with tools to dynamically generate forms. When collecting precise information that’s hard to share by voice, the agent can spontaneously display a form with one or multiple input fields alongside the widget. The user can fill the form or even collaborate with the agent using their voice.

How we built it

We created this library as an extension to the ElevenLabs Conversational AI Widget. It’s a single JavaScript file hosted on a CDN that can be added to any HTML page via a <script> tag. It uses no external vendors or backend. The script provides the necessary client tool implementations, while the reusable system prompt and tool descriptions (available in the GitHub repo) guide the agent on how to use them.

Accomplishments that we're proud of

  1. Created a powerful, fully functional open-source tool that eliminates friction in using voice interfaces facilitating their wider adoption.
  2. None of us had experience with conversational AI or voice agents before the hackathon, so building and publishing a working product in just two days is an accomplishment in itself.
  3. Despite a limited time-zone overlap, we collaborated effectively and delivered quickly, even though it was our first hackathon as a team.

Challenges we ran into

  1. Creating a comprehensive yet concise demo.
  2. Lack of a CLI or IaC tool for agent configuration (requiring us to build our own script).
  3. The agent tends to provide excessive information, despite instructions to be as concise as possible.

Future development

Future development will be guided by community feedback. If there is interest, the project will continue to evolve based on the needs and insights shared by the community. If ElevenLabs sees potential in such hybrid interfaces, this functionality could become part of the standard widget distribution.

Team

Vlad Severin

  • Brainstorming ideas.
  • Solution technical design.
  • Research use cases for the demo.
  • Solution implementation.
  • Project overview.
  • Devtools implementation.

Bohdan Petryshyn

  • Brainstorming ideas.
  • Solution technical design.
  • Research use cases for the demo.
  • Solution implementation.
  • Demo (preparations and recording).
  • Hosting.

Yevhenii Medviediev

  • Brainstorming ideas.
  • Solution technical design.
  • Research use cases for the demo.
  • Solution implementation.
  • Technical documentation.

Compliance

  • The solution idea was brainstormed, designed, and implemented from scratch during the hackathon.
  • No dependencies or paid solutions were used.
  • A three-member team across three different time zones (Ukraine, Poland, Indonesia) worked on the project. We coordinated using Poland’s (CET) time. We started working on the project at Saturday 22 8am (CET)

Built With

Share this project:

Updates