Deep Learning: Natural Language Processing with Transformers

Explore tensor creation with TensorFlow, covering 0D to 4D tensors, casting between data types, and practical tips on shape, rank, size, and converting NumPy arrays.

Explore tensor indexing in TensorFlow, covering 1D, 2D, and 3D indexing, slicing with start:end, negative indices, and steps to select values and subranges.

Explore core linear algebra operations in TensorFlow, including matrix multiplication, transpose, and insum, with practical examples on 2d and 3d tensors, dot and outer products, and batching.

Learn how ragged tensors handle non-rectangular data in TensorFlow, create ragged tensors, apply boolean masks, and use row splits and row lengths to manage irregular sequences.

Learn how to use TensorFlow sparse tensors to efficiently represent tensors with many zeros by specifying indices, values, and dense shape, and convert with sparse to dense.

Explore how TensorFlow handles strings with the strings module, creating 1D string tensors, performing join and split operations, measuring length, and converting inputs to strings.

Learn how TensorFlow variables fix type and shape with an initial value, then update weights using assign, assign_add, and assign_sub, considering broadcasting.

Explore building a linear regression model to predict car prices from features like horsepower, kilometers, and rating, following the machine learning development lifecycle from task definition to training and evaluation.

Prepare data for car price prediction by loading the Kaggle dataset, inspecting features, and splitting inputs and outputs. Shuffle and normalize using TensorFlow to adapt to data.

Learn to build a simple linear regression model in TensorFlow, tune weights and biases for data, and use normalization and dense layers in a sequential model.

The lecture explains how stochastic gradient descent updates model parameters to reduce loss, uses learning rate, and compares SGD with Adam, illustrating training across epochs and convergence.

Learn how to measure performance with root mean square error to compare models A and B, incorporate the metric in TensorFlow model compilation, and evaluate end-of-training performance.

Split data into train, validation, and testing sets to evaluate unseen data. Normalize with train statistics only, use validation split, and compare predictions to actual values using model.predict.

Explore corrective measures for underfitting by diagnosing training and validation losses, and expand model depth with nonlinear activations like ReLU to improve performance.

Explore sentiment analysis by converting reviews into numeric vectors through standardization, tokenization, and encoding, using the IMDB dataset and TensorFlow datasets for binary classification with train, validation, and test splits.

This lecture on standardization explains preprocessing text before numeric modeling, including lowercase conversion, HTML tag removal, punctuation stripping, and comparing stemming versus lemmatization.

Explore tokenization strategies for processing text into models, including character, word, subword software tokenization, and n-gram approaches, with vocabulary, IDs, and one-hot encoding.

Explore tf-idf, which weights terms by term frequency and inverse document frequency, improving bag-of-words representations by emphasizing meaningful words over common ones.

Model sequential data with recurrent neural networks using embeddings and time-step hidden states, and explore bidirectional variants and their streaming limitations.

Master data preparation for sentiment analysis by tokenizing and numerically encoding text with TensorFlow's text vectorization layer, adapting a 10,000-word vocabulary and a 200-token sequence length.

Explore building and training RNNs with TensorFlow using the simple RNN layer, detailing units, embeddings, input shapes, return sequences, and a binary classification training workflow.

Explore why long sequence RNNs suffer vanishing and exploding gradients and short-term memory issues, and learn how LSTM and GRU gates improve sentiment analysis performance.

Learn how word2vec converts words to dense embeddings, compare one-hot vs embedding layers, and train CBOW and skip-gram models with negative sampling to create transferable word representations for NLP tasks.

Integrate a pretrained word2vec embedding (10,000 by 300) from a 3 million word vocabulary via Gensim into a sentiment analysis model, trainable or frozen.

Visualize model embeddings with TensorBoard projector, projecting 300-dimensional vectors to three dimensions and exploring nearest neighbors using metadata and cosine similarity or euclidean distance.

Build, train, and test a sequence-to-sequence translation model with an encoder and decoder. Use embedding layers, a bidirectional gru encoder, a unidirectional decoder, and a softmax over the vocabulary.

Discover how the blue score provides a more reliable evaluation metric for machine translation, including end-gram variants and the brevity penalty to balance length.

Develop and implement a custom bleu score metric from scratch, converting targets to vectors, computing matches with a boolean mask, and integrating it into training in graph and eager modes.

Explore how Bahdanau attention overcomes fixed-length bottlenecks by linking each output to all input positions with learned attention weights. See how bidirectional encoders and additive scoring create context-aware translations.

Build and test Bahdanau attention by implementing the encoder with embedding and the attention layer, compute attention weights to a context vector, and decode with teacher forcing.

Discover the transformer, a simple architecture built on attention alone, with encoder-decoder, self-attention, and multi-head attention, enabling parallel training and superior translation.

Build a custom self attention layer and a custom multi head attention for transformers from scratch, including masking, softmax, and the query key value flow.

Explore visualizing transformer attention by computing and plotting attention scores from softmax across eight heads, revealing how each word attends to others in encoder–decoder contexts.

Use a transformer encoder for sentiment analysis by flattening its output and feeding a single sigmoid dense layer, keeping data processing the same and achieving ~85% accuracy on movie reviews.