Your cart is empty.
Keep shopping
Teach on Udemy
Turn what you know into an opportunity and reach millions around the world.
Learn MoreMastering Generative Voice AI: From Tokens to Agentic TTS
Mastering Generative Voice AI: From Tokens to Agentic TTS
Master SpeechLMs, neural audio codecs, diffusion & flow matching to build real-time agentic voice AI systems
Hot & New
New
Created byVinit Singh
Last updated 7/2026
English
What you'll learn
- Explain the physics, phonetics, and acoustic features that underlie human speech production
- Compare traditional cascade TTS pipelines with modern Speech Language Model (SpeechLM) architectures
- Build and apply neural audio codecs and semantic tokenization (EnCodec, HuBERT, wav2vec 2.0, RVQ) (
- Implement autoregressive codec-based TTS with multi-stream token decoding strategies
- Design unified speech-text models with cross-modal alignment and paralinguistic control
- Apply latent diffusion and conditional flow matching to generate high-quality mel-spectrograms
- Evaluate flow matching vs. diffusion trade-offs for speed, quality, and controllability
- Deploy low-latency, streaming agentic TTS systems with real-time interruption handling
Explore related topics
Course content
7 sections • 374 lectures • 32h 58m total length
- Intro to Lect 1.1 Physics of speech1:14
- 1111 Sound Waves3:05
- 1112 Key Properties of Sound Waves - Amplitude2:19
- 1113 Key Properties of Sound Waves - Frequency5:16
- 1121 Representing & Visualizing Sound Digitally Waveform Spectrum3:22
- 1122 Representing & Visualizing Sound Digitally Spectograms1:43
- 1123 Representing & Visualizing Sound Digitally- Other Representations2:29
- 1131 Applications in Deep Learning2:41
- 1132 Challenges and Considerations1:59
- 1133 SpeechLM Solutions3:45
- 1134 Summary & Key Takeaways1:13
- Coding Example 1.15:34
- Intro to Lect 1.2 The Source-Filter Model of Speech Production1:32
- 1211 The Source-Filter Model of Speech Production - Introduction2:40
- 1212 Components of the Model - 1.The Source2:28
- 1213 Components of the Model - 2.The Filter -Vocal Tract2:07
- 1221 Speech Output1:57
- 1222 Key Concepts of Speech1:08
- 1223 Relevance to Speech Processing1:23
- 1224 Challenges and Considerations2:47
- 1225 Summary & Key Takeaways2:05
- Intro Lect 1.3 Phonetics & Phonology2:35
- 1311 Phones, Phonemes, and Allophones2:41
- 1312 Phonetics and Phonology in Speech1:56
- 1313 Phonetics The Study of Speech Sounds1:52
- 1314 Phonetic Features2:14
- 1315 Phonology the Sound System of a Language2:41
- 1321 Mapping Sounds to Phonemes and Phonetic Features3:25
- 1322 Applications in Deep Learning2:34
- 1331 Challenges and Considerations3:10
- 1332 Summary & Key Takeaways1:47
- Coding Example 1.35:29
- Intro Lect 1.4 Acoustic features1:27
- 1411 Audio Feature Extraction - Introduction1:08
- 1412 Traditional Feature Extraction Mel Frequency Cepstral Coefficients - MFCCs2:38
- 1413 Modern Approaches in SpeechLMs - Raw Waveforms3:38
- 1421 Modern Approaches in SpeechLMs - Learned Audio Representations2:09
- 1422 Comparison- Traditional Feature Extraction vs Modern Approaches in SpeechLM3:30
- 1423 Challenges and Considerations3:47
- 1424 Summary & Key Takeaways1:57
- Coding example 1.4 Acoustic features6:29
- Intro Lect 2.1 The Paradigm Shift: Traditional Cascade vs. SpeechLMs3:05
- 2111 The Paradigm Shift – Traditional Cascade vs. SpeechLMs10:34
- 2112 Theoretical Foundations – Modeling ? ?-? in TTS7:14
- 2121 SpeechLM Paradigm Discrete Tokenization6:54
- 2122 SpeechLM – Discrete Acoustic Tokens & Language Modeling4:43
- 2123 SpeechLM vs Traditional Cascade - Key Advantages5:45
- 2131 Core Concepts – The Cascade Architecture & Its Bottleneck6:35
- 2132 SpeechLM Solution – Discrete Tokenization2:00
- 2133 Training Difference Cascade vs SpeechLM3:09
- 2134 Cascade Approach - Tacotron 2 + WaveGlow – Step‑by‑Step Working3:33
- 2135 SpeechLM – Step‑by‑Step Working3:05
- 2141 Key Models – Classic Cascade Tacotron 2 + WaveGlow7:19
- 2142 The Modern SpeechLM – AudioLM & VALL-E7:29
- 2151 SpeechLMs – Theoretical Challenges & Open Problems5:09
- 2152 SpeechLMs – Research Focus3:25
- 2153 Evolution of Speech Synthesis – From Splicing Tapes to Language Models3:33
- 2154 Future Theoretical Directions – Bridging Discrete & Continuous4:25
- Code Eg 2.1 The Paradigm Shift: Traditional Cascade vs. SpeechLMs10:28
- Intro Lect 2.2 Self-Supervised Learning (SSL) for Speech: wav2vec 2.0 & HuBERT5:04
- 2211 Self-Supervised Learning -SSL for Audio Representation5:03
- 2212 Core Concepts – The Why- The Data Bottleneck2:56
- 2213 Self-Supervised Learning SSL – Theoretical Foundations7:37
- 2221 Pretext Tasks – The Engine of SSL6:34
- 2222 Contrastive Learning – Distinguishing Correct from Distractors7:12
- 2223 Masked Prediction – Learning Acoustic Context by Filling the Gaps4:03
- 2224 Pseudo‑Labeling – Turning Continuous Audio into Discrete Targets4:39
- 2231 SSL Audio Representation – Detailed Working Theory6:31
- 2232 Why The 4 step SSL Audio Representation Pipeline Works2:37
- 2241 wav2vec 2.0 -Contrastive Learning over Quantized Targets4:00
- 2242 HuBERT – Masked Prediction of Hidden Units3:37
- 2243 HuBERT Tokens as Acoustic Vocabulary1:33
- 2251 SSL for Audio – Theoretical Challenges & Open Problems3:58
- 2252 Evolution of Audio Representation – From MFCCs to Self‑Supervised SpeechLMs1:49
- 2253 Future Theoretical Directions – Unified Multimodal & Advanced Discretizatio6:48
- Code Eg 2.2 (SSL) for Audio Representation: wav2vec 2.0 & HuBERT10:54
- Intro Lect 2.3 Semantic Tokenization: Extracting Meaning via K-Means3:46
- 2311 Semantic Tokenization – Extracting Meaning from Continuous Speech7:47
- 2312 Untangling Complex speech The mani-fold hypothesis1:52
- 2313 Core Concepts – Hidden States as Continuous Latents2:04
- 2321 Vector Quantization VQ – Continuous to Discrete Leap3:51
- 2322 K‑Means Clustering for Discretization – Building the Audio Dictionary2:03
- 2323 Limitations & Advances Static centroids , E2E Differentiability2:40
- 2331 Semantic vs. Acoustic Tokens – The What vs. The How4:04
- 2332 Semantic Tokenization – Detailed Working Theory1:50
- 2333 Semantic Tokenization – Centroid Update, Inference & Sequence Collapse1:56
- 2341 wav2vec 2.0 - Online Quantization6:42
- 2342 HuBERT – Offline Clustering for Semantic Tokenization7:53
- 2351 Theoretical Challenges & Open Problems – Semantic Tokenization4:35
- 2352 Evolution of Semantic Tokenization – From GMM‑HMM to SpeechLMs3:31
- 2353 Future Theoretical Directions – End‑to‑End Codebooks & Joint Text‑Audio Spa5:21
- Intro Lect 2.4 Neural Codecs & Tokenization: EnCodec, SoundStream, DAC, and RVQ3:01
- 2411 Neural Audio Codecs & Acoustic Tokenization – The How of Speech8:25
- 2412 Theoretical Foundations of Neural Audio Codecs – Acoustic Tokenization2:43
- 2413 Neural Audio Codecs – Codebook Size vs. Fidelity Trade‑off4:17
- 2421 Acoustic Tokenization – From Audio Slice to Discrete Integers5:04
- 2422 RVQ for Hierarchial Tokenization3:11
- 2423 Core Theory of Audio Representation & Flow – Residual Vector Quantization R2:27
- 2424 The Setup - Raw Audio to Down-sampled Continuous Vectors5:29
- 2425 Continuous Vectors to Residual Vector Quantisation RVQ Codebooks5:38
- 2431 Hierarchical Property of RVQ2:31
- 2432 Residual Vector Quantization – The House Address Analogy2:15
- 2433 Why RVQ wins1:49
- 2441 Google-s Soundstream5:09
- 2442 Encodec -Meta4:19
- 2443 Descript Audio Model DAC4:32
- 2451 Theoretical Challenges & Open Problems – RVQ-Based Neural Codecs3:08
- 2452 Hierarchical Dependency – Unresolved Problems4:49
- 2453 Evolution & Historical Context – From LPC to RVQ Audio Codecs3:10
- 2454 Future Theoretical Directions – Non‑Autoregressive Latent Diffusion & Disen5:03
- Code Eg 2.4 i14:19
- Code Eg 2.4 ii14:48
- Intro Lect 2.5 Decoding & Vocoding: Translating Tokens Back to Audio4:59
- 2511 Decoding & Vocoding – From Discrete Tokens to Continuous Audio8:34
- 2512 Decoding & Vocoding – Theoretical Foundations of Acoustic Reconstruction4:02
- 2513 Decoding & Vocoding – The Generator’s Objective & Upsampling4:52
- 2521 Core Concepts – RVQ Summation From Discrete Indices to Continuous Latent3:56
- 2522 Neural Upsampling – Transposed Convolutions for Resolution Expansion2:09
- 2523 The Phase Reconstruction Problem – From Guessing Phase to Retaining It5:40
- 2531 Detailed Working Theory – Translating RVQ Tokens Back to Audio1:10
- 2532 Decoding & Vocoding – Temporal Dilation & Adversarial Critique2:37
- 2541 HiFi‑GAN- The Legacy Standard5:34
- 2542 EnCodec by Meta – Decoder Architecture with LSTM for Temporal Coherence5:01
- 2543 DAC - Descript Audio Codec – Snake Activations for High‑Frequency Fidelity4:38
- 2561 Theoretical Challenges & Open Problems – RVQ Codec Decoding4:54
- 2562 Evolution of Audio Decoding – From Parametric Vocoders to Codec Decoders2:26
- 2563 Future Theoretical Directions – Latent Diffusion Decoders5:47
- Code Eg i RVQ to audio8:06
- Code Eg ii13:06
- Intro 3.1 Multi-stream token stratgies: Flattening vs. interleavng RVQ codebooks3:28
- 3111 Multi-Stream Token Strategies – Flattening vs. Interleaving RVQ Codebooks8:06
- 3112 Comparison of Strategies - Flattening vs. Interleaving6:22
- 3121 Theoretical Foundations – Factorizing the 2D Audio Token Grid4:13
- 3122 Core Concepts – Semantic vs. Acoustic Tokens7:37
- 3131 Token Flattening – Brute‑Force Serialization of the RVQ Grid3:02
- 3132 Pros & Cons Token Flattening – Brute‑Force Serialization5:59
- 3141 Interleaving - The Delay Pattern – Parallel Depth Prediction w Causal Shift4:11
- 3142 Use in Modern SpeechLMs3:16
- 3151 Detailed Working Theory – Step 1 The Semantic Prior -The Conductor2:44
- 3152 Step 2 & 3 – Flattening vs. Interleaving RVQ Grids3:34
- 3161 AudioLM – Hierarchical Cascaded Flattening - The Pioneer8:35
- 3162 VALL‑E – AR - NAR Splitting - Hybrid Strategy5:39
- 3163 MusicGen – Interleaved Delay Pattern - Unified SpeechLM5:11
- 3171 Theoretical Challenges & Open Problems – RVQ Multi‑Stream Tokenization4:24
- 3172 Evolution of Sound Sequencing – From Continuous Regression to Interleaved R9:36
- Code Eg 3.1-Multi-stream token strategies: Flattening vs. interleaving RVQ codeb9:47
- Intro Lect 3.2 Autoregressive decoding: Sampling, top-k, and cfg for audio4:15
- 3211 Autoregressive Decoding – Sampling, Top‑k, and Classifier‑Free Guidance for9:42
- 3212 Temperature Scaling – Controlling Acoustic Randomness6:07
- 3213 Top‑k and Nucleus -Top‑p Sampling – Filtering the Long Tail2:58
- 3214 Nucleus -Top‑p Sampling - Adaptive truncation5:14
- 3215 Classifier‑Free Guidance (CFG) for Audio – Steering Acoustic Generation6:17
- 3216 Detailed Working Theory – The SpeechLM Decoding Cycle3:33
- 3221 Key Models & Architectures – Decoding Strategies in SpeechLMs3:43
- 3222 Theoretical Challenges & Open Problems – Decoding in SpeechLMs8:33
- 3223 Evolution & Historical Context – Decoding Strategies in Speech AI2:52
- 3224 Future Theoretical Directions – Beyond Static Decoding9:00
- Code Eg 3.2-Autoregressive decoding: Sampling, top-k, and cfg7:51
- Intro Lect 3.3 Speaker conditioning, emotion control, and prosody injection3:09
- 3311 Speaker Conditioning, Emotion Control & Prosody Injection6:27
- 3312 Conditioning Methods & their Comparison6:10
- 3321 Speaker Conditioning, Emotion Control, Prosody Injection – Theoretical Foun4:17
- 3322 Core Concepts – Speaker Embeddings and Acoustic Prompts5:05
- 3331 Prosody - What is it & its type7:12
- 3332 Classifier‑Free Guidance -CFG for Style – Amplifying Emotion & Speaker Iden3:49
- 3341 Speaker & Emotion Conditioning – Detailed Working Theory -Steps 1 to 3 of 51:37
- 3342 Speaker & Emotion Conditioning – Self‑Attention & Sequential Generation -St3:16
- 3351 VALL‑E by Microsoft8:19
- 3352 Spear-TTS by Google4:45
- 3353 Global Style Tokens -GST Legacy – Continuous Emotion Embeddings for Granula5:26
- 3361 Theoretical Challenges & Open Problems – Expressive Speech Control7:33
- 3362 Evolution of Voice Conditioning – From HMMs to In‑Context SpeechLMs2:49
- 3363 Future Theoretical Directions – Factorized Codecs & Unsupervised Emotion Di4:26
- Code Eg 3.3-Speaker conditioning, emotion control, and prosody injection19:53
- Intro to Lect 4.1 Expanding LLM vocab : Joint text–audio tokenization strategies4:41
- 4111 Expanding LLM Vocabularies – Joint Text-Audio Tokenization12:21
- 4121 Key Models Using Joint Tokenization & their Evolution7:05
- 4122 Theoretical Foundations – Joint Tokenization for SpeechLMs3:01
- 4123 Core Concepts – Discrete Text Tokens3:30
- 4124 Audio Tokens – Semantic vs. Acoustic4:26
- 4131 How Joint Text‑Audio Vocabularies Work Together in SpeechLMs3:12
- 4132 The Joint Vocabulary Space – Unifying Text and Audio Tokens –All the steps2:14
- 4133 Vocabulary Expansion & Cross-Modal Interleaving3:03
- 4141 Traditional Cascaded Systems vs. Speech Language Models6:36
- 4142 AudioPaLM & Unified Architectures – Joint Vocabulary with Frozen Text Prior10:11
- 4143 AudioPaLM vs Traditional Cascaded Systems2:27
- 4151 Theoretical Challenges & Open Problems – Joint Text‑Audio Tokenization5:01
- 4152 Evolution & Historical Context – From HMMs to SpeechLMs2:40
- 4153 Future Theoretical Directions – Beyond Discrete Joint Tokenization7:58
- Code Eg 4.1 Expanding LLM vocabularies: Joint text–audio tokenization strategies14:37
- Intro to Lect 4.2 Cross-modal alignment and joint speech–text pre-training objec5:46
- 4211 Cross-Modal Alignment & Joint Speech–Text Pre-training8:31
- 4212 Cross-Modal Alignment – Theoretical Foundations4:57
- 4221 Contrastive Representation Learning for Modality Alignment - InfoNCE loss3:54
- 4222 Contrastive Representation Learning for Modality Alignment - Applications2:30
- 4231 Vector Quantization (VQ) & Discretization – Bridging Speech and Text via Co3:39
- 4232 CTC – Connectionist Temporal Classification A Probability Framework for Sh5:13
- 4241 Joint Speech‑Text Pre‑training – Detailed Working Theory Steps 1‑22:28
- 4242 Joint Speech‑Text Pre‑training – Steps 3‑4 InfoNCE + Joint Masked Predictio2:36
- 4251 Early Fusion vs. Late Fusion for Cross‑Modal Alignment6:18
- 4261 Key Models – Discrete SpeechLMs - AudioLM, VALL‑E1:44
- 4271 Continuous Flow‑Matching & Diffusion Models – ODE‑Guided Speech Generation7:07
- 4281 Theoretical Challenges and Open Problems5:16
- 4282 Evolution of Cross‑Modal Speech Alignment – From Cascaded HMMs to Joint Pre2:46
- 4283 Future Theoretical Directions – Unified Continuous‑Discrete Topologies & An3:55
- Code Eg 4.2 Cross-modal alignment and joint speech–text pre-training objectives19:35
- Intro Lec 4.3 Modeling paralinguistics: Laughter, breathing, disfluency & affect3:31
- 4311 Modeling Paralinguistics – Laughter, Breathing, Disfluency & Affect14:01
- 4312 Paralinguistic Modeling – Theoretical Foundations5:00
- 4321 Disentanglement Theory -Orthogonal Latent Spaces4:14
- 4331 Micro‑Prosody vs. Macro‑Prosody – Temporal Scales of Paralinguistics5:57
- 4341 Disfluency as a Pragmatic Marker – From Noise to Strategic Signal6:47
- 4351 Acoustic Discretization & Audio Tokens – Turning Paralinguistics into Pseud4:51
- 4352 Joint Autoregressive Prediction & Latent Conditioning – Generating Paraling2:57
- 4361 Detailed Working Theory – Modeling Paralinguistics in SpeechLMs3:45
- 4371 Global Style Tokens -GSTs for Macro‑Prosody5:16
- 4372 Hierarchical Variational Autoencoders -VAEs – Nested Latent Spaces for Mult3:33
- 4373 Hierarchical VAEs - The ELBO Method & The Reparameterization Trick7:33
- 4374 Discrete SpeechLMs – Unifying Linguistics & Paralinguistics via Acoustic To6:01
- 4375 Continuous Flow‑Matching & Diffusion – Warping the Probability Flow with Pa9:00
- 4381 Theoretical Challenges & Open Problems – Paralinguistic Modelin8:22
- 4382 Evolution of Paralinguistic Modeling – From Hand‑Crafted Rules to Joint Gen2:09
- 4383 Future Theoretical Directions – Agentic Spontaneity & Multimodal Affect Int3:50
- Code Eg 4.3 Modeling paralinguistics: Laughter, breathing, disfluency, and affec16:34
- Code Eg 4.3 ii Hierarichal VAE & Discrete Prosody-esv2-50p-bg-10p-music-10:51
- Intro 5.1 DDPM, score-based, and latent diffusion architectures for mel-spectrgm2:49
- 5111 Probabilistic Diffusion Mechanics – The Core Intuition6:18
- 5112 Probabilistic Diffusion Mechanics - Mathematical Introduction6:38
- 5113 Denoising Diffusion Probabilistic Model5:47
- 5114 Closed form Sampling ( Reparametrization trick)6:05
- 5121 Forward Process – The Mathematical Foundation6:10
- 5122 Forward Process – Role & Values of Scheduler2:43
- 5123 Reparametrization Trick2:27
- 5124 The Reverse Process – Learning to Denoise3:47
- 5125 Training steps & Objectives2:25
- 5126 U-Net Architecture : The Learning Engine2:38
- 5127 How the Model Learns – Training and Generation4:01
- Code Eg 5.1 part 115:08
- Code Ex 5.1 part 14:55
- 5131 The Compute Bottleneck – Why Pixel Space Diffusion Is Impractical6:01
- 5132 The Latency challenge - What makes it impractical5:56
- 5133 Latent Diffusion Models – Solving the Compute Bottleneck4:55
- 5134 The Spatial compression - Why makes it the right solution3:03
- 5135 Latent Diffusion Models – Training & Generation method2:22
- 5136 Diffusion in the Latent Space – The Practical Revolution6:27
- 5137 LDM: Compute Advantage & Latency Gain6:46
- 5141 Conditioning (Text-to-Image) – Steering the Denoising Process8:16
- 5142 Cross Attention : Application & Implication2:01
- 5143 Cross Attention : Integration in U-net architecture2:23
- 5144 The Complete LDM Pipeline with Cross Attention Integration3:05
- Code Eg 5.1 part 220:53
- Code Ex 5.1 part 215:19
- 5151 Detailed Working Theory – Acoustic Diffusion for TTS3:47
- 5152 Detailed Working Theory – Iterative Denoising in Acoustic Diffusion4:47
- 5161 Denoising Diffusion Probabilistic Models -DDPMs4:28
- 5162 Score-Based Generative Models SGMs5:25
- 5163 Latent Diffusion Models LDMs3:45
- 5171 Theoretical Challenges & Open Problems – Diffusion for Speech AI4:46
- 5172 Evolution & Historical Context – Diffusion Models in Speech AI5:17
- 5173 Future Theoretical Directions – Flow Matching, Rectified Flows & Discrete D7:41
- Code Eg 5.1 part 37:56
- Intro 5.2 Bridging discrete LLM semantic tokens w continuous acoustic diffusion4:54
- 5211 Hybrid Speech Synthesis – Discrete LLM Semantic Tokens & Continuous Acousti8:19
- 5212 Theoretical Foundations – Bridging Discrete LLM Tokens with Continuous Acou5:30
- 5221 Core Concepts – The Information Bottleneck and Semantic Tokens2:55
- 5222 The Latent Acoustic Manifold – Compressing Continuous Waveforms for Tractab6:36
- 5223 Bridging Discrete LLM Semantic Tokens & Continuous Acoustic Diffusion – A H2:14
- 5224 Detailed Working Theory – Hybrid LLM & Diffusion for Speech Synthesis3:17
- 5225 Hybrid Speech Synthesis – Conditionally Guided Denoising & Latent Decoding1:43
- 5231 AudioLDM 2 - LLM & Latent Diffusion Hybrid4:12
- 5232 NaturalSpeech 2 & 3 – Continuous Diffusion with Disentangled Factorized Lat4:37
- 5234 TorToiSe-TTS – The Evolutionary Precursor of Hybrid Speech Synthesis3:30
- 5241 Theoretical Challenges & Open Problems – Bridging Discrete LLM & Continuous5:20
- 5242 Evolution & Historical Context – The Pendulum of Speech Generation3:21
- 5243 Future Theoretical Directions – Beyond Cascaded Hybrids3:36
- Code Eg 5.2 Bridging discrete LLM semantic tokens with continuous acoustic diffu11:23
- Intro to 6.1 Continuous vs. discrete time steps, ODE solvers, and Rectified Flow3:52
- 6111 Continuous-Time Diffusion & ODE Solvers7:47
- 6112 Discrete Time Steps – The Classical DDPM Markov Chain8:48
- 6113 Continuous Time Formulation – From Discrete Steps to a Smooth Flow8:01
- 6114 The Advantages of Continuous Time – Breaking Free from Discrete Steps4:00
- Code Eg 1 Continuous-Time Diffusion & ODE Solvers6:46
- 6121 Deep Dive into ODE Solvers – Probability Flow ODE6:58
- 6122 Comparing Popular ODE Solvers – Speed vs. Accuracy2:00
- 6123 Eulers Method 1st Order – Predictor‑Corrector for Curved Paths3:24
- 6124 Heun's Method 2nd Order – Predictor‑Corrector for Curved Paths3:23
- 6125 Higher‑Order & Adaptive Solvers – DPM‑Solver, DOPRI, and Beyond6:05
- 6126 Continuous vs Discrete – Head‑to‑Head Comparison3:05
- Code Eg 2 ODE Solvers ,Video Models & Rectified Flow7:21
- Intro 6.2 Flow matching vs. diffusion: Quality, speed, and controllability trade4:54
- 6211 Why Diffusion's Geometry Matters6:17
- 6212 Flow Matching & Rectified Flow An introducti3:36
- 6213 The Bottleneck – Why Curved Paths Kill Speed4:00
- 6214 Latency Compute & Performance comparison2:55
- 6215 What we learned about Diffusion steps count1:15
- 6221 Flow Matching The Broader Framework6:43
- 6222 Rectified Flow – The Straight Line to Faster3:28
- 6223 Rectified Flow – Impact on Latency2:43
- 6224 Rectified Flow – Evolution & Industry Adopti2:09
- 6225 The Mathematical Elegance of Rectified Flow1:47
- 6226 Rectified Flow – The Velocity Prediction Obj5:15
- 6227 Why Straight Lines Win – Euler Sampling3:45
- Code Eg 6.2-Flow matching vs. diffusion: Quality, speed, & controllability tra7:54
- Intro 6.3 Conditional flow matching for TTS: Text-to-mel trajectory learning3:18
- 6311 Conditional Flow Matching for TTS – Text-to-Mel Trajectory Learning9:21
- 6312 Theoretical Foundations – Flow Matching for TTS6:38
- 6321 Core Concepts – The Probability Density Path and Vector Fields7:07
- 6322 Conditioning the Flow - The C in CFM6:34
- 6331 Optimal Transport & Trajectory Straightening – The Breakthrough of CFM5:26
- 6332 Detailed Working Theory – Conditional Flow Matching for TTS3:42
- 6341 Matcha-TTS Conditional Flow Matching for Speech2:56
- 6342 Voicebox – Continuous Flow Components for Masked & Conditional Generation5:57
- 6351 Theoretical Challenges & Open Problems – Conditional Flow Matching - CFM6:37
- 6352 Evolution & Historical Context – Text-to-Mel Trajectory Modeling2:46
- 6353 Future Theoretical Directions – Flow Matching for TTS4:38
- Code Eg 6.3-Conditional flow matching for TTS: Text-to-mel trajectory learning6:55
- Intro 7.1 End-to-end multimodal architectures: Eliminating the cascade bottlenec3:59
- 7111 End-to-End Multimodal Architectures – Eliminating the Cascade Bottleneck10:46
- 7112 Theoretical Foundations – The Cascades Conditional Independence Assumption3:44
- 7113 Theoretical Shift – Direct Joint Distribution Modeling2:15
- 7114 Core Concepts – The Cascade Bottleneck and Error Compounding3:37
- 7115 Core Concepts – Latency Accumulation and Native Audio Modality3:41
- 7121 Detailed Working Theory – Acoustic Tokenization & Joint Latent Representati5:34
- 7122 Step 3 & 4 – Autoregressive Audio Generation & Waveform Reconstruction3:56
- 7131 Generative Spoken Language Modeling - GSLM5:46
- 7132 Native Multimodal Transformers – The Unified Pinnacle e.g., GPT‑4o4:40
- 7141 Theoretical Challenges & Open Problems – End-to-End Multimodal Architecture6:31
- 7142 Evolution & Historical Context – The Journey Toward Total Architectural Uni3:30
- 7143 Future Theoretical Directions – End-to-End Multimodal Architectures7:34
- Code Eg 7.1 End-to-end multimodal architectures: Eliminating the cascade bottlen14:21
- Intro 7.2 Zero-shot voice cloning and dynamic emotion transfer in native agents4:24
- 7211 Zero‑Shot Voice Cloning & Dynamic Emotion Transfer in Native Agents11:54
- 7212 Theoretical Foundations – Conditional Disentanglement & Information Bottlen3:44
- 7213 The Information Bottleneck for Zero‑Shot Generalization3:52
- 7214 Core Concepts – The Building Blocks of Expressive Native Agents6:53
- 7221 Detailed Working Theory – Zero‑Shot Voice Cloning & Emotion Transfer - Step6:12
- 7222 Step 4, 5 & 6 – Conditional Fusion, Acoustic Realization & Vocoding6:34
- 7231 Traditional Speaker Adaptation vs. Speech Language Models5:17
- 7232 Global Style Tokens -GST and Variational Autoencoders -VAE8:45
- 7233 Modern Autoregressive & Diffusion-Based Cloners – The State of the Art10:50
- 7241 Theoretical Challenges & Open Problems – Zero‑Shot Voice Cloning & Emotion4:37
- 7242 Evolution of Voice Modeling – From Rigid Statistics to Fluid, Context‑Aware6:54
- 7243 Future Theoretical Directions – Expressive Voice Cloning5:20
- Code Eg 7.2 Zero-shot voice cloning and dynamic emotion transfer in native agent14:01
- Intro 7.3 Minimizing synthesis latency: Chunked inference and speculative decodi2:59
- 7311 Minimizing Synthesis Latency – Chunked Inference & Speculative Decoding16:31
- 7312 Theoretical Foundations – The Chain Rule Bottleneck6:03
- 7313 The Core Insight Behind Speculative Decoding6:53
- 7314 The Mathematics of Localized Context Windows – Linearizing Attention Comple6:27
- 7321 Core Concepts – TTFB, Streaming Bitrate & DraftTarget Models5:22
- 7322 Rejection Sampling & Distribution Alignment – Guaranteeing Quality in Specu4:55
- 7323 Key‑Value - KV Caching in Streaming Frames – Preserving Acoustic Continuity4:28
- 7324 State Caching and Stitching – Smoothing Acoustic Boundaries in Streaming TT7:54
- 7331 Traditional Monolithic TTS vs. Streaming SpeechLMs – The Paradigm Shift6:49
- 7332 Streaming Latent Diffusion & Flow Matching – Sliding-Window Temporal Genera7:50
- 7341 Theoretical Challenges & Open Problems – Streaming & Speculative Decoding5:15
- 7342 Evolution of RT Speech Synthesis – From Phoneme Splicing to Streaming Speec3:18
- 7343 Future Theoretical Directions – Low-Latency Speech Synthesis5:04
- Code Eg 7.3 Minimizing synthesis latency: Chunked inference and speculative deco15:49
- Intro 7.4 Deployment patterns: WebSocket streamng, interruption handlng, & turn-3:28
- 7411 Deployment Patterns – WebSocket Streaming, Interruption Handling & Turn-Tak13:32
- 7412 Foundations of Deployment – Full‑Duplex & Finite State Machines13:17
- 7421 Core Concepts – Persistent Bidirectional Streaming6:29
- 7422 Voice Activity Detection (VAD) and Endpointing – The Turn‑Taking Trigge10:13
- 7423 Barge-In and Interruption Handling – The Acoustic Pivot7:04
- 7424 Turn-Taking Dynamics & Backchanneling – The Dance of Dialogue6:44
- 7431 Detailed Working Theory – The Dynamic Conversational Turn5:25
- 7432 Barge-In – Interruption, Reversion, and Floor Yielding6:34
- 7441 The Half-Duplex Model1:21
- 7442 The Full-Duplex Native Agent – The Architecture of Fluid Conversation3:10
- 7451 Theoretical Challenges & Open Problems – Real‑Time Deployment7:16
- 7452 Evolution & Historical Context – Voice Deployment Architectures2:13
- 7453 Future Theoretical Directions – Anticipatory Turn-Taking4:32
- Code Eg 7.4 Deployment patterns: WebSocket streaming, interruption handling, and12:53
Requirements
- A solid understanding of deep learning fundamentals (neural networks, backpropagation, and training basics)
- Working knowledge of Python and a deep learning framework such as PyTorch
- Basic familiarity with core NLP or LLM concepts (tokenization, transformers, attention) is helpful but not mandatory — key ideas are reviewed in the course
- No prior audio signal processing experience needed — Module 1 builds this from first principles
Description
Generative voice AI has moved far beyond simple text-to-speech — and this course takes you from the physics of sound all the way to building production-grade, agentic voice systems.
Most TTS courses stop at basic vocoders or off-the-shelf APIs. This one goes deeper. You'll start with the fundamentals of human speech — acoustics, phonetics, and prosody — before diving into the architectures actually powering today's state-of-the-art voice models: self-supervised representation learning (wav2vec 2.0, HuBERT), neural audio codecs (EnCodec, SoundStream, DAC), and the tokenization strategies that let LLMs "speak."
From there, you'll master the two dominant modern paradigms — autoregressive codec-based TTS and latent diffusion / conditional flow matching — understanding exactly when and why each is used in real systems. You'll also explore unified speech-text models, paralinguistic modeling (laughter, breathing, affect), and zero-shot voice cloning.
By the final module, you'll understand how to build low-latency, streaming, agentic voice pipelines — the same techniques behind real-time conversational AI agents — covering chunked inference, speculative decoding, WebSocket streaming, and turn-taking.
What you'll learn:
The science of speech production and acoustic feature extraction
How neural audio codecs and semantic tokenization work
Autoregressive and diffusion/flow-based TTS architectures
Cross-modal speech-text alignment techniques
Building low-latency, interruption-aware conversational voice agents
Whether you're an ML engineer, researcher, or voice-tech founder, this course gives you the complete architectural picture — from tokens to agents.
Who this course is for:
- ML/AI engineers who want to move beyond calling TTS APIs and understand how state-of-the-art voice models actually work under the hood
- Speech and NLP researchers looking to bridge classical signal processing with modern generative modeling (diffusion, flow matching, SpeechLMs)
- Voice-tech founders and product engineers building conversational AI agents who need to make informed architecture decisions
- Graduate students or self-taught ML practitioners seeking a rigorous, end-to-end curriculum on generative audio, from acoustic theory to agentic deployment