Thetacast

Stream Globally, Translate Instantly with ThetaCast

Comment 1

Inspiration

Over the last couple of years, we have met up with many people in the community, both at Thetacon and outside of Thetacon. During those meetings we have found out there are people who have a rough time understanding and/or Thetacon or other livestreams especially if English is not their primary language.

What it does

Thetacast is a livestreaming service designed to make conferences more accessible to people who are non-native speakers. It transcribes the livestream and auto translates it into selectable languages, such as Spanish.

How we built it

A Suite of several python tools to directly interface with Theta Video API live-streaming. The URL is also shared with our transcription tool which does the following:

It downloads the stream

Differentiates between speakers

Transcribes it

Translates it into the languages specified

The viewer can then watch the livestream on their screen and select the language they prefer.

Challenges we ran into

Getting the livestream parsed by the transcription tool. It needs to be converted to a different format to get it to work with diart and just using ffmpeg wasn't enough. In the end we got it working and now we’re thinking about improving it and making a pull request upstream on their GitHub so others don't have to write something similar.

Accomplishments that we're proud of

Getting the livestream parsed and converted. And of course: getting the tool to run.

What we learned

how to use ffmpeg in our sleep.

What's next for Thetacast

Thetacon24

Thetacast will make its debut at Thetacon24, held in Vegas this year. We hope to make the conference more accessible to the international community or to those who can't listen to the conference

Admin page

We want to be able to easily tweak some aspects on the fly. This includes selecting more languages, or perhaps allow a way for people to select a language on their own. Right now the languages have to be selected beforehand, all options on Google Translate are possible.

We'd like a way to add a name to a speaker from the Admin page. so everybody would be able to read the correct name associated with the text.

Additionally, this is where the Web3 features would tie in.

Web3

We want to integrate several Web3 features:

  • NFT Airdrop: conference organizers can airdrop NFTs to all online attendees as a proof-of-attendance. These NFTs can be used for the following:

    • Access to TVA DRM-controlled videos
    • Other benefits, for example vouchers for discounts or access to other content

*Automatically raffle NFTs to attendees. This would allow artists to raffle of one of their NFTs from their main collection to the attendees, potentially only to people who also hold an NFT from a different collection.

Interaction

We want to make watching conferences more interactive for those at home.

Q&A

We want to allow online attendees to submit their questions in their native language. These questions could then be addressed at the conference. This ensures the online attendees have an option to have their voices heard.

Interactive polls

We want to add polls, allowing the attendees to vote just like the people in public.

Technical improvements

Stream handling

We want to improve the way the stream gets handled, by streaming directly to the tool and letting the tool stream it to the Theta Video Api. This would allow us to leverage the Theta Video API-DRM which is currently not possible.

Transcription and speaker recognition

Additionally, we will be improving the transcription and hopefully find a way to recognize speakers between streams. At this moment, the transcription hallucinates sometimes, and we want to find a way to improve this.

We'd like to add the option for speakers to give a list of terms they may use, so they can be added to Whisper. This would mean those terms are correctly written in the transcription, which in turn improves the translation.

Customizability

We want to make it easy for the organizers and us to tweak what's needed for the conference as well as make the design of the streaming page align with the branding and color scheme of the event.

Long term

We want to improve on this project. We are very interested in hearing what you think we could improve on or add

Built With

+ 21 more
Share this project:

Updates