Deep Learning Image Generation with GANs & Stable Diffusion

Explore how the variational autoencoder encodes images into mu and sigma, uses a reparameterization trick, and minimizes KL divergence with the prior to reconstruct or generate images.

Build and train a variational autoencoder on MNIST, with an encoder outputting mu and log var and a decoder reconstructing 28x28 digits, then sample z to generate new digits.

Visualize the latent space of a variational autoencoder by plotting two-dimensional Z vectors colored by digits, revealing digit-wise clustering, and compare gradient-tape training with Keras fit using an overridden trainstep.

Explore how generative adversarial networks train a generator and discriminator to produce realistic images, with applications in anime characters, super resolution, and text-to-image generation.

Master the gan loss and training setup by examining the discriminator and generator, real versus fake data, and the minimax game driving convergence in deep convolutional gan.

Explore stabilizing GAN training in TensorFlow by adopting binary cross entropy loss, modifying architecture with strided convolutions and batch norm, using leaky relu, noisy labels, and other practical tricks.

Train a gan to generate celebrity face images by alternating a generator and discriminator, using a celeb faces dataset, dcgan architecture, a 100-dim latent, and 64×64 color images.

Study the Wasserstein GAN with a critic and the Wasserstein loss to stabilize training. See how 1-Lipschitz enforcement via weight clipping and more critic iterations boost stability and output quality.

Improve training of Wasserstein GANs by comparing weight clipping and gradient penalty, showing gradient penalty preserves higher moments and stabilizes critic gradient norms without batch normalization.

Implement wgan in practice by removing batch normalization in the discriminator, expanding its capacity, and applying gradient penalty to train the critic with multiple updates.

Explore progressive growing of GANs (ProGANs) to generate high-quality images from 4x4 to 1024x1024 by simultaneously growing the generator and the discriminator, improving stability and training speed.

This lecture demonstrates implementing ProGAN practices in TensorFlow, including progressive growth from 4x4 to 128x128, pixel normalization, equalized learning rate, mini-batch std, and fade-in to stabilized discriminator and generator.

Demonstrates photorealistic single-image super resolution using a GAN with perceptual content loss (MSE plus VGG19 features) to push outputs toward the natural image manifold, outperforming MSE-only approaches.

Train an srgan end-to-end by downloading a 30,000-file dataset, preprocessing to 16 by 16 and 64 by 64 resolutions, and optimizing with vg loss and content losses.

Explains diffusion models that generate high quality images from text prompts by gradually adding and removing noise, using beta and alpha parameters, and predicting the noise with a neural network.

Build a unet model for diffusion-based image generation, denoising via time-step conditioning. Prepare a 64x64x3 Kaggle dataset, normalize to -1..1, and train with skip connections.

Describe how a time step is converted into an eight-dimensional positional embedding and integrated into residual blocks with jell-o activation and group normalization in a diffusion model.

Explore how attention layers link every word or pixel to all others, using query, key, and value mechanisms to assign weights and enable self-attention in language and image generation.

Train a gpm diffusion model by sampling time steps, creating noisy images from real data, and predicting the noise with mean square error loss and gradient tape.

Demonstrate diffusion sampling by starting from random noise x_t, denoising with alpha, beta, and sigma, and iterating from T to 1 using the model output.

Explore how diffusion models transform noise into images from input prompts, train a U-Net to predict noise, and apply latent diffusion with text conditioning for interior design outputs.

Use the diffusers library to run RunwayML's Stable Diffusion 1.5 in an image-to-image pipeline, adjusting strength, guidance scale, and steps to balance prompt fidelity and preserving the initial image.

Explore an inpainting pipeline using segmentation masks, depth maps, positive and negative prompts, and a control net with stable diffusion to modify only chosen regions.

Explore zero-shot depth estimation with a depth anything model to automatically generate depth maps from images using a dedicated depth map pipeline, tested on house images and other interiors.

obtain segmentation maps and masks from an image using grounding dino and the segment anything model, by labeling walls, floor, ceiling, windows, and doors.

Explore generating interior scenes with stable diffusion using control nets, depth maps, and masks; tune prompts, negative prompts, and conditioning scales to produce varied, photorealistic designs.