I think it is time to bring that paperclip back

Hack4Fun: Use STT, LLM, TTS to create a chatbot and launch a Twitch livestream of the chatbot interacting with chat or even VRChat players. Work mostly consists of connecting available services.

Comment 1

Inspiration

The 24/7 Twitchbot streamers such as ask_jesus, Chat_with_AI and others. A popular chatbot can then prioritize messages that include donations in the queue.

Links to the 24/7 Twitch streams:

Another inspiration, this sad sketch of Clippy: https://www.youtube.com/watch?v=ooP49OR8q0U

What it does

The user speaks/writes to the chatbot. An Local Language Model prepares a clever/smug response. Text-To-Speech engine converts the text into an audio samples and plays it back, possibly in the late Gilbert Gottfried's voice. The process continues indefinitely. Hopefully leaving the speakers with all around fun conversations.

If the development team is large enough, we stream the result to Twitch.

How we built it

AI Models required for this are already available free and deployable locally, it is only a matter of connecting the services.

The project's scope depends on the final development team size in order to for the 48 hour timeframe:

  • 2-3 members - Interaction with the chatbot would be done through a website, accessible even on phones.
  • 4 members - Attempt to run the bot as an assistant in a Hub world of VRChat. It hears the players and responds to their queries.
  • 5 members - Run a Twitch stream, bot responds to Twitch chat.

VRChat is a social hub, where interaction is mostly with 3D avatars and voice. It can be run without VR headsets and even on Android phones. Only if there are at least 4 team members would this integration be even attempted. The major challenge with VRChat integration would be splitting up the incoming player audio. Hopefully there is a way to intercept them before they all get merged as one audio output.

Team required:

  • Someone finding the most suitable LLM and prompt (Task can be done without coding knowledge)
  • Someone to find or create a high quality TTS voice clone and optionally use STT/ASR for voice input rather than text (English, primarily)
  • Someone to create the API that connects all previous services (chatbot itself)
  • Someone to deploy the chatbot as VRChat Avatar which also accept voice queries from the game and then send the TTS produced audio back through the PC's microphone.
  • A 3D artist/animator capable of working with simple face animation (Viseme, Blender)

Help, is coming to you soon:

Built With

Share this project:

Updates