Captions Sync: AI Recognition

Inspiration

Sometimes I think about vlogging in a video + captions format, but existing solutions are very expensive and don't always cover what is needed. So I decided to create my own application, with the right subtitle generation algorithms.

What it does

Imagine an app that transforms the way you connect with the world. Something simple, yet powerful. An app that understands your content, your message, and your audience.

I've made sure that Captions Sync isn’t just smart, but intuitive. You can customize keywords, adjust the tone, and even switch between languages with ease. It's as flexible as you need it to be, but powerful enough to give you exactly what you're looking for.

It seamlessly connects to your social media platforms, making sharing as easy as tapping a button.

In the end, Captions Sync is more than just an app. It’s a tool that amplifies your voice, sharpens your message, and elevates your content to new heights. It’s about empowering you to share your ideas, your passions, your story with the world in the most impactful way possible.

How we built it

I've only used pure native techniques. SwiftUI for layout, SwiftData for storage data and Metal for loading animations. SFSpeechRecognizer is key for recognition, but I left room for editing subtitles that weren't recognized correctly. The generation algorithms are based on lower-level technologies, CALayer (and its variations), AVKit, and so on.

Challenges we ran into

I used low-level techniques to generate subtitles on CATextLayer, CALayer and AVMergeComposition. The big challenge was the arrangement of the words (and the separate animations for each). This is a problem in many analogs, as they either highlight all at once, or the timings are incorrect, and sometimes the words pop out somewhere else.

So the main difficulty was in sequential animations and optimizing to avoid crashes. It took me about 10 days (out of all the way) to get the subtitles to display correctly (and choose their position).

By the way, you can use any font, border highlighting and more.

Accomplishments that we're proud of

I found it difficult to use the algorithms because not everything always worked as it should (either the text went out of bounds, or there were difficulties with timings). But everything seemed to work out. I published the whole story here: https://twitter.com/Viktorianec.

I'm glad it worked out to do full customization of the subtitle position, make locale selection, and give a decent amount of customization options in a short amount of time.

I'm very happy that the algorithms are working as they should! You should definitely try the effects out!

What we learned

Reacquainted with low-level APIs, working with video and a huge amount of animations. I am glad that RevenueCat acted as a stimulus for such a project. It was a big challenge for me to force myself to get it to the point of putting it in the store.

What's next for Captions Sync

Of course, there's still a lot to improve. We need more subtitle effects, more time to devote to AI for generating translations and maybe sounds. But the most important thing is that the display algorithm is already working

Built With

• avkit
• caanimation
• metal
• revenuecat
• swiftdata
• swiftui