Data Science: Transformers for Natural Language Processing

Discover where to get the course code and resources, including notebook code via the resources tab Code link, and GitHub materials, with tips to avoid common retrieval mistakes.

Navigate this course by level, beginner, intermediate, and advanced, with titles indicating difficulty, and use practice, experience, and Q&A to progress at your own pace.

Ask questions via Q&A to get answers within 24 hours and deepen understanding, stay anonymous, meet prerequisites, and engage with handwritten notes for conceptual lectures and coding along.

Learn to handle a 403 error when downloading from lazy programmers by manually downloading the file in a browser and uploading it via the Colab file explorer using drag-and-drop.

Enter the beginner's corner for transformers and learn to use them with the Hugging Face interface, exploring sentiment analysis, embeddings, named entity recognition, text generation, translation, and zero-shot classification.

Explore how transformers build on RNNs with attention to capture long-term dependencies, using encoder-decoder architectures, bidirectional context, and parallelized computations to translate sequences efficiently.

Explore sentiment analysis as a binary classifier and learn to use the Hugging Face pipeline to perform it with one line of code, including interpreting label and score.

Learn to perform sentiment analysis with transformers in python by using the hugging face pipeline, filtering neutral labels, and evaluating with accuracy, f1, and auc on a tweets dataset.

Frame language as a time series to predict the next word with autoregressive and transformer models, using hugging face pipelines to generate text from prompts and explore practical applications.

Explore text generation in python using transformers and a GP2 pipeline to generate poetry text. Observe how prompts, seeding, and multiple samples expose the model's limitations and creative potential.

Explore bidirectional language modeling with masked tokens, contrasting Bert's bidirectional encoder with autoregressive GPT, and see how article spinning uses fill-mask predictions in transformers.

Spin BBC news articles using a masked language model in Hugging Face with a fill mask pipeline in Python, including automatic word replacement.

Learn to perform named entity recognition with transformers, identify people, places, and companies, using IOB tagging and aggregation to group multi-token entities, and inspect per-entity scores and spans.

Learn named entity recognition with transformers using a pipeline, load the Carnell 2003 dataset, and evaluate the test set with accuracy and F1.

Explore text summarization with transformers and learn to build a Python pipeline to summarize text. Distinguish extractive from abstract summaries and see practical applications like abstracts and executive summaries.

Explore text summarization with transformers, using the BBC news dataset and a Distilled BART pipeline to generate abstractive summaries via an encoder–decoder model.

Explore neural machine translation as a sequence-to-sequence task using ends and transformers, with input and target text pairs, and implement translation in Python with the pipeline API and blue score.

Explore neural machine translation with transformers by loading an English–Spanish dataset, preparing tokens, translating 1000 phrases, and evaluating results with the blue score.

Explore extractive question answering, where a model identifies a contiguous answer span within a context, using squad benchmarks and a simple question context input format.

Learn how to perform question answering in Python with a pre-trained Transformers pipeline, using a no-dataset demo to create context and questions, test synonym handling, and extract answers via indexing.

Master zero-shot classification in NLP, leveraging a pipeline to classify text against arbitrary candidate labels, producing a scored, label-ordered output that adapts to any class set.

Explore zero-shot classification in python with transformers, using the BBC News dataset on GPU, and classify documents into labels like biology, math, geology, politics, and tech.

Explore how transformers address sentiment analysis with context, generate text and embeddings for similarity search, and perform tasks like named entity recognition, summarization, translation, question answering, and zero-shot classification.

Learn to submit structured feedback through a simple suggestion box to help tailor the data science and natural language processing course, from background and course difficulty to requested topics.

Explore fine-tuning transformer models for natural language processing, from tokenization and pipelines to training with Hugging Face, with practical sentiment analysis and multi-input sentence classification examples.

Review text pre processing for neural networks, including tokenization, token-to-index mapping, and dynamic padding, with emphasis on word, character, and sub word tokenization and truncation in transformers.

Explore how the hugging face pipeline preprocesses text, applies a model, and post-processes outputs, using auto tokenizers and models to create ready-to-use inputs in PyTorch.

Learn to install transformers and use the auto tokenizer in Python, tokenize hello world, and examine input IDs, attention masks, and special tokens for sequence classification.

Explore transfer learning with transformers, using pre-trained networks as feature extractors and fine-tuning final layers to tackle image and text tasks in NLP, computer vision, and beyond.

Clarifies transfer learning vs fine tuning in nlp, detailing how pre-trained weights adapt to new tasks, with unfreezing strategies and bias-term fine tuning for efficiency.

Learn how self-supervised pre-training builds language models that predict the next token or fill in masked words, powering GPT and Bert styles and driving transfer learning.

Fine-tune a pre-trained text classifier on a sentiment analysis dataset from the glue benchmark using hugging face datasets and trainer, then save and deploy the model with a pipeline.

Fine-tune transformer models for sentiment analysis in python by loading transformers and datasets, tokenizing data, training with a trainer, and evaluating with accuracy on the SST-2 task.

Fine-tune transformers on a custom airline tweet dataset for sentiment analysis, from loading data and mapping labels to integers to train/test splits, tokenization, trainer, accuracy, F1, imbalance, and confusion matrix.

Use hugging face auto config to set custom label mappings. Load from pre-trained checkpoint with auto config, update label-to-id mappings, and run a pipeline yielding clear labels like positive.

Explore fine-tuning transformers for textual entailment with two-sentence inputs, using Bert and DistilBert, including tokenization, special tokens, and binary classification for entailment tasks.

Fine-tune transformers for text classification with input sentences using the RTC dataset from glue benchmark. Train with the distilled checkpoint, tokenize two sentences, and evaluate using accuracy and F1.

Fine-tune a pre-trained transformer to boost text classification on your data, mastering tokenization, preprocessing, padding, and training via pipelines and Hugging Face Datasets.

Explore token classification for natural language processing by turning per-document labels into per-token predictions using transformers and the hugging face library, covering data processing, tokenization alignment, and sequence-level metrics.

Explore data for named entity recognition, with input sentences and target tags. Showcase a flexible dataset with tokens, any tags, POS tags, and chunk tags, plus tokenization options.

Load and inspect the data set, set up the tokenizer, and tokenize data while mapping label ids to their names for train, validation, and test splits.

Realign and expand word-level labels to match subword tokenization, handling bert special tokens and -100 for ignored positions with a custom alignment algorithm using word IDs.

Create a tokenized dataset by aligning labels with tokenized inputs, applying tokenize function in a map, and removing columns to produce input ids, attention masks, token type ids, and labels.

Explore implementing online label alignment for subword tokens in transformer NLP, mapping begin to inside tags, handling -100 tokens, and aligning labels during batch tokenization.

See how the data collator handles per-batch padding and converting tokenized inputs into PyTorch tensors behind the trainer, especially for token classification tasks, with options to define a custom collator.

Explore creating and using a data collator for token classification: convert tokenized data to per-sample dictionaries, handle padding with -100 labels, and prepare tensors for loss.

Compute token classification metrics for multiple targets using the Sequel library with sequences of sequences, handling -100 padding and IAB format tags to produce accuracy, precision, recall, and F1.

Explore evaluating sequence labeling models by computing accuracy, precision, recall, and f1 on batched label sequences, handling -100 paddings, per-entity metrics, and converting labels to strings; implement a metrics function.

Load a model from a checkpoint and fine-tune it on the dataset using the token classification auto model with explicit id-to-label mappings, then train and save the model.

Load a pretrained token classification model from a checkpoint and set id2label and label2id mappings. Fine tune with training arguments and weight decay, then save and test via a pipeline.

Learn token classification with a custom dataset by converting the Brown Corpus into JSON inputs and targets for POS tagging. Load into Hugging Face and measure accuracy and F1.

Develop a complete pos tagging workflow on a custom dataset with transformers, including data prep, tokenization, label alignment, training, evaluation, and deployment checks with sample outputs.

Extend token classification by labeling tokens for named entity recognition and parts of speech tagging, loading Hugging Face dataset, tokenizing, fine-tuning a model, and saving and reloading as a pipeline.

Introduce the fine-tuning section for transformers in neural machine translation, covering two-language tokenization, data preparation, and the transition to seq2seq models and the trainer.

Load the Katie dataset for English to French translation, subsample 1000 pairs, and prepare data with tokenization using a pretrained checkpoint, including input and target tokenization with a context manager.

This field moves fast, with changes and breaks in libraries such as tokenization; adopt the text target approach and learn any library version to adapt quickly.

Install transformers data sets and sentencepiece, load an English–French dataset, sample 1,000 for quick train‑test preparation, then tokenize inputs and targets to compare correct and incorrect tokenization.

Explore how sequence-to-sequence models use an encoder and a decoder to translate sentences. Train on input sentences and corresponding target sentences and condition the decoder on encoder outputs.

Tokenize the inputs and targets, assign the targets to the labels field, and use a data collider to pad sequences and convert them into tensors for a seq-to-seq translation model.

This lecture guides building a seq2seq translation pipeline by analyzing input and target lengths, setting max lengths, tokenizing, and configuring a data collator and decoder input IDs.

Learn to evaluate translations with bleu score and bert score using prepared code: install the sacre bleu and bird score libraries, load the metrics, and compute predictions against targets.

Compare translation metrics BLEU score and BERT score, install and load metric libraries, decode predictions, and understand how contextual embeddings and cosine similarity improve meaning over exact word matches.

Configure the training arguments and trainer to train and evaluate a fine-tuned model, evaluating before and after training; save and load it as a translation pipeline with generate predictions.

Train and evaluate a transformer for English-to-French translation in data science: transformers for natural language processing, using autoregressive generation and fp16 precision.

Learn neural machine translation as a seq2seq task with encoder–decoder models, tokenize inputs and targets, and evaluate using blue and bird scores for practical training.

Fine-tune transformers for extractive question answering by processing context and question pairs. Tokenize inputs, compute metrics, convert model outputs to text, train and evaluate, and handle Hugging Face IDs.

Explore the squad data set, the Stanford question answering data set, and learn extractive question answering where answers are substrings of the context, with questions, contexts, and answer lists.

Install and use transformers and datasets to load the SQuAD dataset, explore train and validation splits, and examine fields like id, title, context, question, and answers, including multiple answers.

Tokenize the question and context with bert or distilbert, using the course tokenizer, and handle long contexts by overlapping windows with a stride, overflow to sample mapping, and offset mappings.

Learn to tokenize with a pre-trained model, using auto tokenize, and split a long context into windows with max length and stride; inspect and decode input ids and offset mappings.

Align the targets after splitting the context into multiple windows by recomputing the start and end indices for the answer using sequence IDs, offset mapping, and token positions to form training samples.

Align the targets in Python by converting character positions to token positions within context windows, using sequence IDs and offset mappings to generate neural network targets for transformer inputs.

Implement and apply a tokenized function to train and validation samples using map, pad to 384 with 128 stride, and manage offset mappings and sample IDs for start/end token positions.

Define tokenized train and validation functions to process data, strip whitespace, use offset mappings to align answers, and pad to a 384 max length with a 128 stride.

Learn to compute question answering metrics using the squad benchmark, returning exact match and f1 scores from predictions and references, including multiple possible answers.

Learn to use the squad metric in Python by loading the metric, preparing predictions and references with IDs, and computing exact match and F1 scores on dummy data.

Convert model outputs into readable answers using a pre-trained question answering model, decoding start and end logits with context windows and offset mappings for evaluation.

Extract text answers from logits by applying transformers for natural language processing, using a fine-tuned squad model, tokenization, and Python-based inference on the GPU.

Develop a manual compute metrics function that derives best answers from start logics and end logics using offset mappings, then call metric compute to produce results.

Define and run a dedicated compute metrics function for transformers in NLP, processing the dataset with context windows, sorting logits, and selecting the best answer before computing metrics.

Train and evaluate a pre-trained auto model for question answering by configuring a trainer and training arguments, running training, computing metrics on validation data, then saving and deploying via pipeline.

Extend training and evaluation by loading pretrained checkpoints, fine-tuning the head, and using a trainer to predict on the validation set and compute exact match and F1 scores.

Fine-tune a model for question answering by integrating tokenization, metrics, and trainer pipeline, tackling long context, window limits, and misaligned answers while converting transformer outputs for metrics.

Explore the theory behind transformers for natural language processing, including self-attention, scaled product attention, multi-head attention, and the encoder-decoder architecture, with no coding required.

Explain a basic self-attention mechanism and how the context vector is a weighted sum of input hidden states. Compute attention weights with softmax over dot-product scores.

Learnable parameters extend self-attention by mapping inputs to queries, keys, and values, enabling scaled dot-product attention with softmax to compute attention weights and context vectors.

Explain how self attention handles variable length sequences and enables parallel computation across tokens. Highlight the t^2 cost and why attention remains superior for long sequences.

Explain how attention masking completes self-attention by masking irrelevant tokens, like padding and patch tokens. Reveal that the mask uses minus infinity before softmax to zero out attention.

Explore how multi-head attention extends self-attention with multiple weighted heads, concatenates the heads, and projects back to a single t by d_model representation for parallel, efficient transformers.

Explore the transformer block as the core building block for encoders, decoders, and seq2seq models; add a feedforward with gelu, apply layer normalization, and use skip connections.

Discover how positional encodings inject order into transformer embeddings by adding sine and cosine vectors at multiple frequencies to uniquely identify each position.

Explore how to build a transformer encoder from embeddings and positional encoding through multiple transformer blocks with self-attention to produce contextual embeddings for text and token classification.

Explore how decoder architecture in transformers predicts the next token using masked, causal self-attention, with a final vocabulary classification layer and parallels to the encoder structure.

Learn how to combine an encoder and decoder into a sequence-to-sequence transformer, using attention to connect encoder outputs to the decoder and understanding teacher forcing and inference.

Explain the bert encoder-only transformer, pre-training with masked language modeling and next sentence prediction, and how base and large models compare in performance.

Explore GPT, a decoder-only pre-trained transformer that learns from unlabeled text by predicting the next token, then fine-tunes with two heads for language-modeling and tasks like classification, entailment, and multiple-choice.

Explore GPT-2's larger data and parameters, expanded context, and how prompt formatting achieves translation and summarization without fine-tuning, using the Web text built from Reddit posts.

Explore GPT-3's scale and in-context learning, 175 billion parameters and 2048 context, trained on web text, books, and Wikipedia for zero-shot, one-shot, and few-shot transfer in translation and trivia QA.

Explore how chatgpt, a revolutionary chatbot by OpenAI based on GPT 3.5, generates human-like responses and supports tasks from conversation to translation and content creation.

Explore GPT-4, a multimodal language model handling text and images, with 8,192-token base context and a version with over 32,000 tokens, offering improved alignment and LSAT and bar exam performance.

Explore the theory behind attention and transformers, including self-attention, unsupervised pre-training, encoder-decoder architectures, and model variants like Bert and GPT.

Implement transformers from scratch using PyTorch, building encoder, decoder, and encoder-decoder models without pre-made layers, mastering multi-head attention, positional encoding, and shape mastery.

Plan and build an encoder transformer in PyTorch with multi-head attention, transformer blocks, residual connections, layer norms, and positional encoding, plus embedding and a final classifier, using Hugging Face tokenizers.

Explore the multi-head attention implementation in PyTorch, covering Q, K, and V projections, scaling, masking, and the final projection with reshaping and broadcasting.

Learn to implement the transformer block from scratch by wiring its init with d_model and heads, plus dropout, two layer norms, a multi-head attention layer, and a feed-forward network.

Implement positional encoding from scratch by defining init and forward methods, precomputing max len based sine and cosine terms with a numerically stable div term and applying dropout.

Build a full transformer encoder from scratch by integrating embedding, positional encoding, stacked transformer blocks, layer normalization, and a final linear classifier for text classification.

Train an encoder on a real sentiment dataset with huggingface tokenization and SST from the glue benchmark, using PyTorch data loaders with input IDs and attention masks.

Implement causal self-attention in the decoder with a lower-triangular mask to attend only to past tokens, while integrating padding masks and mirroring the encoder architecture in transformers.

Builds gpt transformer decoder from scratch, detailing embedding, positional encoding, transformer blocks, layer norm, and a final fully connected layer to predict the next token in a many-to-many language model.

Train a causal language model from scratch with transformers and Hugging Face tools, covering tokenization, data loading, padding, next-token prediction, and text generation.

Implement a full encoder-decoder sequence-to-sequence transformer by integrating the encoder and decoder with a configurable causal mask. Test with dummy inputs on gpu to verify cross-attention, masking, and output shapes.

Continue building and training an encoder-decoder transformer for english-spanish translation, including data download, preprocessing, tokenization, dataset preparation, train-test split, and a training loop.

Continue building a seq2seq transformer for language translation by implementing the train method, preparing decoder inputs by shifting targets, creating masks, running forward and backward passes, and demonstrating translation generation.

Implement transformers from scratch by building encoder, decoder, and encoder-decoder models, applying positional encoding, training on data sets, and using the attention mechanism with basic components.

Take a bird's-eye view of large language models and transformers, and explore scaling laws, self-supervised pre-training, supervised fine-tuning, RLHF, and debunk myths about building ChatGPT from scratch.

Differentiate between using and building AI models, align your goals, and switch between builder and user mindsets when using tools like NumPy, PyTorch, and the OpenAI API.

Explore scaling laws that govern how model size, dataset size, and compute jointly drive performance in language models, and learn why balance and data quality matter for efficient NLP.

Transformers attend to all past inputs, enabling parallel processing and better context retention, at the cost of higher time complexity than RNNs.

Foundation models train language models through self-supervised pretraining, predicting the next word with softmax and cross-entropy loss. Explore transformers, scaling laws, and the hardware-software challenges of large language model training.

Learn how language models are aligned to our preferences beyond next-word prediction using supervised fine tuning, reinforcement learning from human feedback, direct preference optimization, and group relative policy optimization.

Explore the impact and usage of language models, from sentiment analysis to summarization, and learn to balance cost, performance, and engineering decisions.

Explore multimodal and visual language models that process image and audio inputs, generate images, and interact through voice, enabling AI to act in the physical world.

Explore how large language models evolve into agentic AI by wrapping an LLM in perception, reasoning, and action loops with tools and memory for goal-directed behavior.