sample result of stream audio transcription used for clip recommendations
To help viewers to find older great channel clips!
What it does
Recommends clips of the same game when watching a stream.
The intent was to real-time transcribe a livestream and search for clips on the spoken text or keywords. We were able to independently record near realtime livestream, transcribe, and query the clips api.
How we built it
The front end extension is a Typescript VueJS app. It also has Typescript NodeJs Extension Backend Service that makes calls to the Twitch API (Helix).
We built a development template here
Challenges we ran into
Working with the Twitch Extension Helper was somewhat difficult. For example, the conditions on which the Context changes is not completely clear. Settings up Typings for Twitch and Twitch Extension Helper was tricky.
Accomplishments that we're proud of
- Vue development toolchain
- Live Twitch stream audio and video frame extraction.
- RxJS based Vue Twitch Extension Helper Plugin.
- JWT server validation.
What we learned
Twitch API, how to develop a Twitch Extension, handling m3u8 streams in python
What's next for Cliptastic
More features and more efficient server processing. Currently the clips have to be listed and filtered in memory, since the API does not support filtering.
Server deployment and extension store publish. After some more testing and minor cleanup, this is ready to be published.
Features we would like to add:
- Default featured clips, configurable by the streamer.
- Remember a viewers watched clips, as to recommend new ones.
- More intelligent suggestions, currently it just lists the top clips for a streamer and filters by game
- Overlay suggestions triggered by transcribed audio.
- Development wise, integrating browsersync for faster frontend development.