A Tutorial on Speaker Diarization

Explore a modular unsupervised framework for speaker diarization, from feature extraction and VAD to segmentation, speaker embeddings, and clustering with online and offline methods.

Explore hierarchical clustering, focusing on agglomerative methods that merge embeddings by highest affinity using various linkages. Learn optimizations with max heaps and canopy clustering to improve efficiency.

Explore spectral clustering for speaker diarization by building an affinity matrix from embeddings, refining it, and using eigenvalue gaps to determine the number of clusters.

Explore why unsupervised clustering falls short for speaker diarization and how supervised methods using training data and temporal information improve clustering, including neural networks like recurrent models and transformers.

Explore supervised speaker diarization with UIS-RNN, PIT/EEND, and VAD, covering speaker change modeling, sequence probability, and end-to-end clustering approaches.

Present a supervised speaker diarization approach using ts-vad and dnc, leveraging target speaker vad, multi-speaker and multichannel models with attention, plus data augmentation methods and viterbi post-processing.

Tackle challenges in speaker diarization, including overlapping speech, input features, and leveraging multiple microphones. Explore future directions with ASR integration, foundation models, end-to-end approaches, and richer transcriptions.

Discover Google's speaker diarization approach using LSTM-based embeddings to segment and label multi-speaker audio, applying spectral clustering with refinement operations for accurate, permutation-invariant evaluation across public datasets.

Explore fully supervised speaker diarization with an online, end-to-end model that combines speaker change, assignment via Chinese restaurant process, and sequence generation for improved clustering.

Discriminative neural clustering (DNC) reframes speaker diarization as a supervised sequence-to-sequence task, using transformer models to map speaker embeddings to cluster labels, with augmentation and curriculum learning enabling differentiable diarization.

Explore Google's turn-to-diarize system, combining transformer transducer based speaker turn detection with constrained spectral clustering to produce efficient, turn-level speaker labels.

Explore word-level end-to-end neural speaker diarization, adding an auxiliary encoder and joint network to a frozen ASR model with shared blank logits.

Explore Google's speaker diarization journey from modularized systems to natural language models, including supervised end-to-end methods and multimodal LLMs.