NLP-Natural Language Processing in Python(Theory & Projects)

Define regular expressions as a language to specify rules for strings in a text corpus, enabling powerful search and replace, and contrasting them with Python string functions.

Discover why regular expressions provide fast, one-line pattern matching in Python for searching and replacing complex patterns, often preferred to string functions, with rare cases needing code.

Explore how Eliza uses pattern matching to respond without understanding. Learn how Python regex patterns drive outputs for phrases like 'I need x' or 'tell me about your family'.

Explore practical regular expressions in Python with the re module, using findall and iterators to locate patterns in strings, and note the theoretical roots in regular languages.

Explore how meta characters power regular expressions in Python, with examples such as dot, dollar, asterisk, plus, question mark, brackets, backslash, pipe, and parentheses.

Explore meta characters and character classes in regular expressions, including ranges and hyphen notation, and apply them to count digits in a string using a focused exercise.

Count digits in a string using a digit character class in Python's regular expressions, showing two patterns: '0-9' and '0123456789', and highlighting that order matters only with a hyphen.

Practice using square brackets as a regex meta character to detect four consecutive digits in a string with Python, returning true if any four-digit sequence exists.

Apply Python regular expressions with square brackets to detect four consecutive digits in a string, using re.find_all to extract patterns like 3534 or 4985 and report found or not found.

Explore the cap meta character in Python regular expressions, learn how placing it at the start of a character class yields a set complement, and practice with coding examples.

apply meta characters cap exercise 3 to filter documents and select those with no digits and no restricted symbols such as colon, quotes, or parentheses.

Explore meta characters in regular expressions using Python to identify documents that do not contain any digits or specified special characters, comparing direct and complemented character classes.

Learn how the backslash acts as a crucial meta character in Python strings and regular expressions, covering escaping, common pitfalls, and counting backslashes in LaTeX documents.

The lecture explores counting backslashes in a document, why the escape character disrupts naive searches, and how three backslashes are needed for correct counting in Python regex.

Explore how backslashes are treated in Python strings and regular expressions, why raw strings matter, and how to count backslashes accurately in a document.

Practice counting backslashes and square brackets in a raw-form document, identifying seven backslashes and two square brackets in the example to report a total of nine occurrences.

Practice solving a regex task in Python by counting backslashes and square brackets in a document, using escaping to treat brackets as ordinary characters, and exploring multiple solutions.

Demonstrate counting backslashes and square brackets in a document using two separate regexes in Python, then print and sum the counts to solve the problem.

This lecture demonstrates counting occurrences of the backslash pattern in LaTeX, using the meta character to find each \section occurrence and understand pattern matching, with multiple solutions possible.

Explore backslash regex techniques and special sequences in Python, solving a backslash section search and counting spaces, digits, and underscores using class shortcuts like \d, \s, and \w.

Count spaces, digits, and underscores in a document using a Python regex class with special sequences like \s and \d, then print matches and the count.

Explore how the asterisk in regular expressions enables repeating patterns from two to many occurrences, illustrated with Yahoo and varying o's. Apply this pattern matching to identify valid variable names.

Explore meta characters and sterics to identify longest valid variable-name substrings in a document, applying rules that names must not start with a digit and may include digits and underscores.

Learn to extract valid variable names from a document using a regular expression that starts with a non digit and allows letters, digits, and underscores.

Learn to validate variable and function names from code by building a list and checking rules: no starting digit and no special characters except underscores, reporting syntax errors.

Explore greedy matching in Python regex, using the star and other metacharacters to match A to B with a class, and how the engine prunes when the final pattern fails.

Explore plus and steric regex meta characters, comparing at least one occurrence to zero or more, and learn how question mark makes patterns optional. Use examples like Yahoo to illustrate.

Explore the curly bracket meta character in regular expressions, defining repetition ranges with lower and upper bounds, and practice extracting digit substrings of length 2–5 from a document.

Demonstrate a Python regex to extract numbers with two to five digits using backslash D and {2,5}, then introduce regex objects and search, replace, and split operations.

Learn pattern objects by compiling regular expressions with re.compile and using methods like findall, search, and substitutions to locate and replace patterns across strings with flags such as ignore case.

Learn about pattern objects in regex, compare the match and search methods, and apply a match-based exercise to count leading space characters at the start of a document in Python.

Learn a Python regex solution to count leading spaces at the start of a document with a pattern object and the match method, using M.span, M.start, M.end, and M.group.

Learn the difference between pattern objects' match and search methods in Python regex, including how match starts at the string, how search finds the first occurrence, and how findall differs.

Explore how finditer returns an iterator of match objects with start, end, and span, enabling location-aware pattern detection for valid variables in a document.

Learn to locate valid variable names with Python's re.finditer on a compiled pattern, extracting spans and groups. Compare finditer and findall, and apply re.IGNORECASE to match across cases.

Master the vertical bar as the logical or to join multiple regular expressions in Python, creating a single pattern that matches brackets or backslashes and counting matches with find all.

Explore meta characters in regex, the hat (^) and dollar sign ($), to anchor patterns at the beginning or end of strings and lines, and understand complements inside character classes.

Explore how opening and closing parentheses form groups in regular expressions to treat multiple characters as a single unit, enabling repetition, pattern matching, and targeted replacements in Python.

Explore string modification with regular expressions by splitting strings at pattern matches into a list and using substitute or subn to replace matches and return a new string with count.

Implement a simple word tokenizer using the split function to extract words by splitting on non alphanumeric characters in Python for text preprocessing and tokenization.

Apply regex-based string replacement in Python to colors like blue, red, and white, substituting with color, collapse multiple spaces, and trim spaces at the ends.

Master text preprocessing by collapsing internal spaces with pattern p1 and trimming leading and trailing spaces with pattern p2 using regular expressions in Python.

Explore what counts as a word in natural language processing, distinguishing words from punctuation and the boundaries between words in spoken versus written text.

Explore the definition of a word in NLP as task dependent and language dependent, and examine word units, punctuations, and disfluencies for speech-to-text and speaker identification.

Define words for a task, then form the vocabulary as the set of unique words (word types) from a corpus, ignoring punctuation.

Learn how tokens differ from word types and how tokenization creates a running list of words. Understand vocabulary, punctuation handling, and upcoming tokenizing with spaCy and regular expressions.

Explore the spaCy package for modern natural language processing in Python, install and load English models, and tokenize text, including punctuation.

Prepare a Yelp review dataset for a sentiment classifier by tokenizing text, building a vocabulary with unknown tokens, and classifying reviews as negative or positive.

Initialize the vocabulary and set up token mappings for the Yelp reviews dataset, load and inspect reviews.csv, and outline an add_token function to avoid duplicates while handling an unknown token.

Learn to build and manage a vocabulary for nlp by implementing add_token and add_many_tokens functions, mapping tokens to indices and expanding the vocabulary as you process documents.

Explore building and using a vocabulary for a Yelp reviews classification mini project by implementing lookup token and lookup index helpers, handling unknown tokens, and initializing the vocabulary from data.

We build a vocabulary from the data frame by tokenizing reviews, counting word frequencies, and selecting tokens above a cutoff, then apply one-hot encoding for representations.

Explore one hot encoding to convert words into vectors for Yelp reviews classification, illustrating vocabulary indices, column and row vectors, and preparing token encoding in Python.

Learn to implement one hot encoding and build a vocab from a generic corpus, accepting numpy arrays or other objects, and create a fixed-size document vector for Yelp reviews.

Practice encoding Yelp reviews with one-hot vectors, averaging them to create a fixed-size document vector for classification, and discuss limitations of losing word order before exploring TF-IDF and RNNs.

Implement a document feature vector by averaging one-hot token vectors, building a numpy feature matrix, and setting up data for Yelp reviews classification.

Explore a Yelp reviews classification pipeline in Python by building a from-scratch vocabulary from training data, performing a train-test split, and generating feature matrices.

Define corpus to feature matrix for x_train to produce a matrix where each row is a document's feature vector, and transpose to meet scikit-learn expectations for training and test data.

Explore faster compute_features and corpus_to_feature_matrix with one-hot and averaged token features for Yelp review classification. Train a logistic regression model and compare with SVM, evaluating with a confusion matrix heatmap.

Explore tokenization and vocabulary building in NLP, then encode words with embeddings, train classifiers, and navigate preprocessing, normalization, and sentence segmentation challenges.

Explore tokenization challenges in natural language processing, from space-based tokenizers and punctuation handling to multiword expressions. Discuss languages with no spaces and handling special characters and context-dependent tokens.

Apply a data-driven tokenization approach with byte pair encoding. Build a language-agnostic vocabulary by merging frequent adjacent character pairs into subword tokens; implement BPE in Python.

Explore how byte pair encoding works as a tokenizer using a toy corpus, merging frequent adjacent character pairs into new tokens and outlining a Python implementation.

Explore byte pair encoding by building a vocabulary through successive merges and use a greedy test-time tokenization to convert unseen words into subword tokens.

Implement byte pair encoding for tokenization by importing numpy and building a get_pair_counts function that computes pair frequencies from a corpus, illustrating symbol-based word splitting and pair counting.

Identify the best pair from counts, merge it into the corpus, and repeat k merges while recording which pair merged at each step for byte-pair encoding.

Implement a byte pair encoding training loop on a corpus, generating merge indices and statistics, updating the corpus, and prepare the encoding of new text using learned BPE rules.

Learn how byte pair encoding tokenizes new words by converting them into underscored characters, generating all adjacent character pairs, and selecting the best pair to merge using BFE statistics.

Identify and merge the most frequent byte pairs using BPE stats to build new tokens. Demonstrate selecting the best pair and applying iterative merges to encode words.

Learn how to implement byte pair encoding for tokenizing new words using BPE statistics, including finding and merging the best pairs to produce tokens.

Explore preprocessing challenges in tokenization and apply case folding to standardize tokens, ensuring similar forms like USA with dots or hyphenated variants are treated consistently.

Explore lemmatization, a normalization technique mapping words to roots. Observe a spaCy demonstration of tokenization and lemma extraction, noting stop words and language dependence, and stemming in the next video.

Explore word normalization through stemming, a naive lemmatization that extracts roots via affix rules. Rely on language-dependent rules to remove ing forms and suffixes, aiding speech recognition and dialogue systems.

Expose students to word normalization and sentence segmentation by applying tokenization, abbreviations handling, and sentence boundary rules with spaCy, then explore spelling correction via a distance algorithm.