Mastering Generative Vision & Video: From GAN to Flow to DiT

Explore probabilistic diffusion mechanics from DDPM to latent diffusion models, detailing forward noise addition, reverse denoising, and how text prompts guide generation via cross attention.

Compare physics diffusion with diffusion models, illustrate forward noise addition from image to Gaussian noise, and explain how a neural network learns the reverse denoising to recover the original image.

Apply the reparameterization trick in diffusion models (DDPM) to predict noise with a u-net, using a fixed forward Markov chain and a learned reverse process.

Explore how diffusion models forward the image by adding noise in fixed stepwise increments toward gaussian noise, using a Markov chain and the beta_t scheduler to define each step's distribution.

Explore the U-Net architecture as the core learning engine, featuring down blocks that compress, a bottleneck, and upsampling with transpose convolutions and skip connections to preserve spatial information.

Train the model to predict the noise in the reverse process, using MSC loss, across epochs, to generate high-quality images from Gaussian noise.

Cross attention uses q from image and k and v from text, applying softmax over sqrt(dk). During diffusion, the focus starts on the main subject and shifts to details.

Apply cross attention in the U-net from the bottleneck to upsampling blocks with text embeddings. Exclude it on downsampling blocks, as pixel-based DDPMs demand more compute than latent diffusion models.

Learn how clip-based embeddings guide diffusion with dynamic cross-attention from bottleneck to up-sampling in an LDM-based model using a text-based prompt, and generate a cat wearing a hat.

Examine how flow matching and rectified flow overcome diffusion model limits, moving from DDPM to SD 3.5 and Flux 1/2 architectures for faster image and video generation.

Explore how traditional diffusion requires thousands of steps and high compute, and how flow matching with rectified flow enables near one-step generation via straight trajectories in forward and reverse processes.

Analyze the bottlenecks of traditional diffusion, focusing on the curved forward and reverse paths. Show how flow matching, rectified flow, and distillation reduce latency and improve quality with fewer steps.

Flow matching trains a continuous vector field to guide gaussian noise toward the final image along a straight path, via rectified flow, reducing steps and making diffusion deterministic.

Showcases rectified flow to dramatically reduce diffusion latency, enabling one-step generation with high quality, or adjustable steps for better quality, while offering a straight-path, deterministic sampling and efficient GPU utilization.

Explore rectified flow using a time parameter t to move from pure noise to real image along a straight line, blending x data (signal) and x noise (noise) via zt.

Flux 2 fuses mistral 3 vlm with a flow transformer to 32 billion total parameters, enabling world knowledge, reasoning, and advanced visual language understanding for mirror reflections and object interactions.

Explore Flux 2.0’s fusion of Mistral visual language model with rectified flow to deliver physics-aware, text-preserving image generation, refraction, and shadow realism.

Explore stable diffusion 3.5 and flux 1,2 models on limited vram by applying 4-bit quantization, dropping heavy encoders, and offloading work to the cpu for four-step generation.

Explore continuous time diffusion and ode solvers, compare ddpm sde and dpm solver, and see how 20-step odes outperform 1000-step discrete approaches in stable diffusion.

Explore how probabilistic flow ODE transforms noisy SDE into a smooth deterministic vector field, enabling fast real-time diffusion and compare Euler, Heun, and DPM solvers for speed versus accuracy.

Explore higher-order and adaptive solvers for diffusion models, comparing DPM solvers with fixed-step Euler and Hewenn, and DOPRI variants to optimize speed, accuracy, and CFG guidance.

Contrast discrete and continuous generative approaches: discrete models are slow and rigid, while continuous probability flow ODEs with flow matching or rectified flow enable fast, high-quality video generation.

Master cross attention to steer a stable diffusion model, create two prompt embeddings, and interpolate them to generate blended images with a DPM solver and CFG weighting.

ip adapter introduces two parallel attention paths—text and image—where the image path uses a reference image to guide generation, weighted by lambda for controllable image influence.

Master IP adapters and decoupled cross attention to control style and subject identity in generative vision. Compare global and grid tokens and explore clip encoder integration and IP adapter variants.

Explore diffusion model guidance with control net and t2i adapters, comparing encoder copying with lightweight pose and style adapters, plus zero convolutional training and ip face id variants.

Explore structure control for diffusion models with ControlNet and T2I adapters, combining text prompts with image-based guidance—pose, edges, or depth—to improve pose accuracy and layout control.

Discover how control nets evolve from u-net to transformer-based dit architectures, using zero-initialized trainable copies, cross-attention injections, and multi-scale conditioning with pose, depth, or edges.

Freeze the pre-trained model and train a zero-initialized copy of its encoder blocks to inject control maps at multiple scales, preserving knowledge while enabling condition-driven generation.

Use zero convolution by initializing weights to zero to preserve base knowledge and gradually inject control during training, avoiding random initialization that could corrupt generation and cause forgetting.

Stack multiple adapters for pose, depth, and sketch; T2i adapters are lightweight and fast, while control net delivers higher accuracy for critical product design tasks.

face id uses an ip adapter with reference images and 512-dimensional embeddings from arcface and adaface to preserve identity. tune lambda to balance identity, clothing, and background.

Explore control net and t2i adapters, focusing on zero convolution, encoder copies, and condition maps for control-driven image generation via edges and depth; compare architectures and fidelity-speed trade-offs.

Hands-on with control net and t2i adapters using canny edge maps from synthetic scenes via OpenCV. Generate images with stable division model and compare outputs for prompts as Mars cyberpunk.

Explore consistency models and LCM distillation for fast, single-step image generation across any time step, and learn latent consistency models and LoRa-based training to boost speed and quality.

The consistency model predicts x0 from any x_t via a parametric f_theta, minimizing mean square error, enabling 1–2 step generation with global manifold awareness and real-time latency.

Explore the consistency models training approach and its loss function, which minimizes f(x_t, t) versus f(x_t', t') to recover the original image in one step, avoiding Euler or Huon.

Explore consistency models through coding examples that demonstrate the consistency property, boundary conditions, and training methods (distillation and isolation), paired with DDPM comparisons and c_skip and c_out dynamics.

Combine LCM with LoRa to create a universal accelerator for peft-based fine-tuning with low compute. Distill models with two matrices and 4-step generation, plug into SD 1.5, SDXL, or NM.

Explore latent consistency models and LoRa acceleration for diffusion models, including training a consistency function, denoising trajectories, and latent space visualization with PCA.

Adversarial diffusion distillation combines a student, a discriminator, and a teacher to generate 1–4 step images, using adversarial and score distillation sampling losses for photorealistic, artifact-free results.

ADD models boost sharpness and texture by avoiding averaging inherent in MSC distillation, outperforming LCM in one-step generation; a hybrid ADD approach combines distillation with adversarial training for fast results.

The add solution uses two masters—a pre-trained original model and a second master discriminator—to combine score distillation and adversarial losses for sharp, semantically accurate, photorealistic image generation at lightning speed.

Explore Flux1, a 12 billion parameter model with two text encoders, enabling language understanding, and dual-stream diffusion transformer using rectified flow to generate images from Gaussian noise in few steps.

Explore autoregressive image generation and StyleGAN3 as alternatives to diffusion. Compare Parti/VAR models, raster tokenization, and anti-aliasing improvements for robust, fast image synthesis.

Explore autoregressive image generation, tokenizing 256×256 images with a vision transformer and vqgan, then generating token sequences left to right via a transformer and reconstructing images with a decoder.

Expose the limitations of autoregressive 1D sequence for 2D images, including slow generation and ignored 2D locality, and introduce parallel VAR models as the next approach.

The lecture contrasts raster scan with token-based generation to VAR models that generate scale maps in parallel, reducing steps from 1024 tokens to about 10–20 and boosting inference speed.

Examine raster-scan autoregressive image generation, including patchification, embedding, vector quantization, and a 64 by 64 latent grid quantized to 8192 tokens via a VAE encoder, with progressive stage reveals.

The lecture explains StyleGAN 1 architecture from Gaussian noise to w latent space and uses disentangled styles across coarse, middle, and fine levels, noting ADAIN brightness spreading as a problem.

StyleGAN 3 overcomes StyleGAN 2 aliasing by adopting continuous feature fields with Fourier features. Enable smoother motion and employ Elias-free continuous space convolutions to manage high-frequency details.

Explore StyleGAN 3's continuous, equivariant design that solves texture sticking and aliasing, enabling hair to move with head rotation and real-time edge-friendly inference.

Explore StyleGAN3 architecture, including equivariance, alias-free operations, and Fourier-based features, to address texture sticking and aliasing in video generation; compare latency and model trade-offs between GAN and diffusion approaches.

Explore how 3D DiT tokenizes video into a 3D tensor, applies self-attention across space-time tokens, and replaces convolution to enable 3D generation.

Explore how diffusion transformers borrow large language model architectures for visual data by predicting noise. Understand 3d tokenization, patchification, and the scaling laws that boost video quality.

Convert 3d video data into 1d tokens using space-time patches and a 3d variational autoencoder, enabling transformers for diffusion-based video generation.

Explore rotary positional encoding for video transformers, enabling relative 3D positioning by decoupling time, height, and width, and applying rotation-based embeddings to video patches.

3d rope extends rotary embeddings to space-time from 1d to 3d, enabling relative understanding across frames. It underpins video dit and tracks objects in scenes with Sora and View.

Explore the 2025-2026 video generation leaders—OpenAI Sora, Google VO2, and Runway Gen3 Alpha—unified by diffusion and diffusion transformer architectures, with varying trade-offs in length and fidelity.

Learn to run 3d spatio-temporal attention DIT diffusion models on 6 GB VRAM in lab environments using spatial windowing, quantization, and distillation, with multi-GPU sharding for consumer GPUs.

The lecture contrasts spatial attention per frame with temporal attention across frames, using a B x F x C x H x W video tensor to ensure consistent video generation.

Explore how optical flow estimates per-pixel motion between frames using motion vectors dx, dy, grounding deep learning models like Raft, FlowNet, and PwCnet in brightness constancy and correlation volumes.

Apply dense optical flow vectors dx and dy to reveal pixel motion, enabling motion estimation, object tracking, and physics-prior guided CGI motion transfer.

Inject optical flow into latent space to warp features and inform denoising, guiding per-pixel motion for realistic, flicker-free, long-range video coherence.

Explore motion buckets that condition video diffusion to control motion from an image, using optical flow dx dy based on UV values to bucket motion from still to extreme.

Explore discrete motion buckets that convert per-frame optical flow into 0-255 integers, mapping ranges to motion levels from subtle to extreme, enhancing controllability and training stability in video diffusion generation.

Control camera motion separately from object motion in diffusion-based video generation using noise augmentation to prevent entangled motion and enable tracking car a while car b moves independently.

Explore optical flow with brightness constancy and physics priors to compute dense motion fields, then apply motion buckets for image-to-video generation with diffusion models.

Explore conditioning video generation with camera motion controls using plucker coordinates, a six-dimensional ray representation, and learn how Klein constraint and flow matching in diffusion models enable realistic, controllable results.

Explore how Euler angles capture camera movement in 3d space by combining camera position with orientation, using pitch, yaw, and roll to describe per-frame motion.

Compute frame-wise Euler angles to derive angular velocity and frame-to-frame panning. Encode per-frame pitch, yaw, and roll into diffusion control with cross-attention in DIT.

Explore how Plucker coordinates replace Euler angles to provide global pixel control, encoding rays with direction and moment vectors via P cross D, with the camera center as origin.

Examine the Plucker relation and Plucker coordinates, detailing the six-dimensional d and m and the zero dot product condition, and explore their use in diffusion-based flow matching.

Explore how a 3D line can be defined by two points or by the intersection of two planes, using Plücker coordinates and related operations.

Explore how differentiable Plücker coordinates enable linear interpolation of camera motion via a constant del L/dt, conditioning the velocity field for rectified flow in video generation.

Explore neural audio synthesis by contrasting discrete and continuous latent space approaches, highlighting AudioLM, MusicGen, and StableAudio 2.0, and how to compress raw waveforms into a latent space.

Compare discrete token models and continuous flow models for audio generation, using latent space vs raw waveform, and examine long-range coherence with transformers or ODE solvers.

The lecture explains converting analog audio into digital form by representing raw waveform with spectrograms and mel spectrograms. It highlights distilling acoustic features and semantic latents for neural networks.

Compress raw waveform to a latent space via an encoder, then decode via a neural codec; compare residual vector quantization and continuous flow approaches for scalable, real-time audio.

Compress eight audio samples into four with a four-sample kernel and stride four, then build a feature map via filters and dot products toward latent space at 50 hertz.

See how a latent vector passes through kernels and convolutional filters, then ReLU activates positives to form the latent space, before mapping to discrete tokens with residual vector quantisation codebooks.

Explore the continuous tokenization approach using flow matching to move from noise to latent audio in a straight-line diffusion path, enabling high-quality audio generation in few steps.

Compare discrete rvq with continuous flow matching, highlighting a differentiable latent space, fast non-autoregressive inference using od solvers, and audio generation via a vae encoder and stable audio 2.

flow matching converts the noise-to-data journey in diffusion into a straight-line path, enabling large steps, reducing truncation error, and producing fast, high-quality audio with discrete and continuous tokenization.

Explore discrete residual vector quantization and continuous flow matching to tokenize audio from raw waveform, using eight code books and an encodec 24 kHz model, then reconstruct and compare quality.

Explore the discrete token based and continuous flow based approaches to neural audio generation, detailing audio lm, music gen, and a two-stage semantic and acoustic process.

Explore MusicGen’s interleaved delayed pattern that bypasses the RVQ bottleneck by delaying deeper codebooks, enabling parallel generation and fast, controllable music synthesis with a single transformer.

Explore current frontiers in generative models: Riemannian flow matching in hyperbolic or spherical latent spaces, discrete-continuous hybrids, and neural operator formulations for music generation.

Learn how discrete-token audio lm and musicgen use RVQ tokenization and encodec to synthesize audio, and compare with stable audio 2.0 through continuous flow matching and ode-based generation.

The course introduces the multimodal diffusion transformer (MM-DiT) architecture for end-to-end audio-video generation, emphasizing temporal alignment, cross-modal attention, and time-based token interleaving for native audio-video synthesis.

Master lip-sync post-processing with wave-to-lip and sync-net two-network system, where sync-net acts as judge and wave-to-lip as artist. Explore architectures, training details, and post-generation uses like dubbing and real-time avatars.

SyncNet acts as the judge of lip-sync accuracy by evaluating video-audio alignment with visual and audio encoders. Wave2Leap complements SyncNet to enhance dubbing and AI avatars in post video editing.

Discover SyncNet’s role in lip-sync training, dubbing quality control, and benchmarking, using a dual-stream video–audio encoder and a cosine sync score. Explore transformer, temporal, and multi-scale variants and contrastive loss.

Explore how SyncNet enforces phoneme-to-viseme precision by matching lip shapes for sounds like b, p, f, v, and m, ensuring crisp articulation of plosives in an adversarial loop.

Learn how FAD uses Wasserstein distance and how Syncnet uses cosine similarity to evaluate latent audio video alignment, including a coding example.

Understand Wasserstein 2 distance, the earth mover distance, and how a gaussian assumption yields FAD score; apply Fretcher distance in latent space with a VGG encoder to gauge audio quality.

Study latent embedding makes fad robust to phase shifts and volume changes, aided by a perceptual encoder, SyncNet uses frames and audio spectrograms with cosine similarity for benchmarking, lip-sync training.