SD | Devpost

Speaker diarization identifies “who spoke when” in multi-speaker recordings and supports applications such as meeting transcription, conversational analysis, and multi-speaker speech recognition. Conventional diarization systems use separate stages for speech detection, segmentation, speaker embedding extraction, and clustering. Although these systems can handle a flexible number of speakers, they often struggle with overlapping speech because each frame is usually assigned to a single speaker. End-to-end neural diarization directly predicts frame-level speaker activity and improves overlap handling. Still, most existing methods rely only on audio and remain sensitive to noise, reverberation, and poor recording conditions. Audio-visual diarization introduces complementary cues from faces and lip movements. Yet, visual information may be missing or unreliable because of occlusion, off-screen speakers, tracking errors, or non-speech lip motion. Existing multimodal approaches commonly combine audio, visual, and speaker features through direct concatenation, which may not ensure correct speaker-level alignment or adapt to changing modality reliability. Therefore, an important research gap is the lack of an identity-aware fusion mechanism that can maintain speaker consistency while dynamically selecting reliable multimodal information.