CaptionMate

Inspiration

As a video editor myself, I sometimes find myself wishing that I could translate my videos that are spoken in English into a different language, such as Spanish or Mandarin, in order to reach a further audience and even to practice my own familiarity with the language. Additionally, most YouTube videos have automatic captioning generated, but this is easier for people to caption their videos themselves and even gives them the option to translate their video transcript into a different language if they'd like.

What it does

This add-on takes a video put into Adobe Express and first transcribes the video's audio into text. The user then has the option to translate the given text into another language, provided that that language is an option.

How we built it

The elements of the add-on's UI were made and developed using Adobe's modified version of React code (React Spectrum and React Aria) and the translation and transcription services were modified from Amazon's initial AWS translation and transcription.

Challenges we ran into

Getting stuck when coding elements of the add-on's UI, dealing with new and unseen before error messages, and learning new technology and software from Amazon's AWS.

Accomplishments that we're proud of

Persevering through our first hackathon, learning and debugging more aspects of React and TypeScript, and being able to connect all of the different parts of our project together.

What we learned

How to use Amazon's AWS transcription and translation services, coding in React and TypeScript, team building, and planning out and finishing a project in a short period of time.

What's next for CaptionMate

Real-time captioning that immediately follows spoken audio in the video instead of producing solely a transcription and translation of the video's spoken audio and possible extensions to captioning and translating solely audio without a video requirement.

Built With

• adobe
• react
• typescript