modes
audio tags
transcription page
timeline page
timeline page dark mode
Dashboard
Legal page
Landing page

Inspiration

Traditional subtitles tell us what was said, but they often leave out everything happening around the speech. For people who are deaf or hard of hearing, a door slamming, someone laughing, crying, shouting, or a change in emotional tone can be just as important to understanding a scene as the dialogue itself.

That made me wonder: what if captions could understand the context of a video instead of simply transcribing its words?

That idea became Auralis.

What I built

Auralis is an AI-powered accessibility system that turns audio and video into richer, context-aware captions.

Instead of generating only dialogue, Auralis can analyze speech, sounds, tone, emotion, emphasis, and visual context. It can identify events such as laughter, crying, or a door slamming and associate them with timestamps.

The user can then choose how much information they want their captions to contain, from standard subtitles to captions that include emotions, environmental sounds, or both.

How I built it

I designed Auralis as a pipeline rather than treating AI as the final output.

The media is first analyzed to extract the dialogue and relevant audio/visual information. Gemini is used as the analysis layer to understand speech, sounds, emotions, emphasis, and context. The resulting information is converted into structured, timestamped data.

Auralis then decides how that information should actually appear to the user. For example, an emotion-aware mode can add emotions in brackets, while a sound-aware mode can describe environmental events such as [door slams] or [laughter].

This separation lets the AI focus on understanding the media while Auralis controls the accessibility experience.

Challenges

One of the biggest challenges was deciding what information should actually be shown.

It is easy to make captions more detailed, but more information does not automatically mean better accessibility. Too many descriptions can overwhelm the viewer, while too little can remove important context.

I therefore focused on making the system modular, allowing different caption modes instead of forcing every user to receive the same amount of information.

Another challenge was getting different types of information to work together while maintaining accurate timestamps and keeping the final captions readable.

What I learned

Building Auralis taught me that accessibility is not simply about adding more technology. It is about understanding what information a person actually needs and presenting it at the right time and in the right form.

I also learned how powerful multimodal AI can be when it is used as an interpretation layer rather than simply as a chatbot. Auralis combines speech, audio events, emotions, and visual context into structured information that software can then turn into an accessible experience.

Most importantly, the project changed how I think about inclusive design: sometimes the biggest accessibility improvement is not creating a new interface, but making sure important information was never hidden in the first place.