Master Natural Language Processing using case studies

Explore natural language processing and text analytics, analyzing text data from social media, insurance claims, banking, and education to extract meaning, detect sentiment, and power chatbots and security, including counter-terrorism.

Learn how to transform raw text into translatable language by breaking data into small problems, and applying lexical, syntactic, and semantic processing for real-world natural language processing tasks.

Apply lexical, syntactic, and semantic processing to convert text into meaningful representations. Learn how word forms, grammar, and word meaning drive applications like spam detection, Q&A, and translation.

learn how text encoding underpins natural language processing, from ASCII origins to Unicode, UTF-8 and UTF-16, and why choosing the right encoding matters for English and non-English text.

Master regular expressions (regex) to search, extract, and substitute patterns in text, using python's re library to find matches, capture groups, and handle hashtags, emails, and phone formats.

Develop skills in regular expressions by analyzing quantifiers—zero or more, zero or one, and plus—and applying a find patterns function to match text with patterns.

Master anchors for start and end pattern checks, wildcards for any character, and whitespace handling in regex; learn grouping, alternation, escaping, and flags for case and multiline matching.

Learn to build regular expressions with character classes, ranges, and negation to match patterns such as usernames, passwords, and alphanumeric inputs, with case sensitivity and whitespace shortcuts.

Explain how regular expressions use a greedy approach to maximize matches, from the minimum two to five, and how a trailing question mark makes it non-greedy.

Master natural language processing with regex techniques, exploring match, sub, find all, and capture groups to extract dates, years, months, and indexes.

Explore regular expressions to filter files by extension, using patterns, anchors, and backslashes to avoid false matches, with practical examples and regex101.com guidance.

Explore stopwords in natural language processing, and learn how removing common words affects text preprocessing, word frequency distributions, and dimensionality in machine learning for text data.

Explore tokenization to split text into words and sentences, handle noise, punctuation, hashtags, and emojis, and transform text into numeric features for models like spam detectors.

Master bag-of-words by tokenization and stop-word removal, convert text to a sparse document-term matrix, and feed it to models like logistic regression for spam detection data.

Learn how to handle similar words in NLP by contrasting rule-based stemming, including Portus and Snowball techniques, with a more intelligent dictionary-based approach that looks up word variants.

Explore how stemming and lemmatization handle similar words, comparing rule-based stemmers like Porter and Snowball with the WordNet lemmatizer, considering part-of-speech effects, speed, and accuracy.

Apply preprocessing to text by lowercasing, tokenizing, removing stop words, and joining tokens to build a bag of words representation, then compare limited edition and stemming options on spam data.

Explore the limitations of bag of words and learn how tf-idf uses inverse document frequency to weigh terms by relevance across documents.

Learn to build a tf-idf model from text in Python, including preprocessing and bag-of-words transformation, and inspect features on a spam dataset.

Construct an end-to-end spam detection case study with text mining, preprocessing, and a base classifier that achieves around 99% training and 98% test accuracy using feature presence vectors.

Learn to identify and process spelling mistakes and variations in text by applying phonetic hashing, minimum edit distance, and multiword expression techniques to improve NLP preprocessing.

Explore the soundex algorithm to bucket misspelled words by phonetic similarity, using consonant-to-digit rules, four-letter codes, and remove duplicates with Delhi and Mumbai as examples.

Develop a function to produce four-letter codes from words like Bombay and Mumbai by uppercasing the first letter, building a dictionary with rules, replacing dots, and discarding duplicates.

Uncover methods to correct spelling mistakes using edit distance, exploring insertions, deletions, and substitutions. Build a practical spell checker and understand string distance for NLP applications.

Learn to compute string distance in python by building a dynamic programming matrix, populating it from matches and nearby cells, and exploring a transposition variant and library comparison.

Explore spell correction as a preprocessing step in natural language processing, implementing NORVIG, tokenization, lowercase normalization, and frequency-based word data to suggest correct words.

Explore a spell corrector that generates one- and two-edit candidates from a base text, splits words, and ranks corrections by frequency-based probability using the Adventures of Sherlock Holmes text.

Develop a spell corrector by using a tokenized world, a probability function, and loanwords filtering; use possible connections to select the most probable word.

Learn to handle combined words like 'New Delhi' using pointwise mutual information (BMI) to decide when to treat as a single unit, via joint and conditional probabilities.

Learn to handle multiword sequences with the bigram language model, simplifying probability calculations for phrases like to be or not to be, and explore tools for PMI and n-gram scores.

Explains syntactic processing as the next step after lexical processing. Covers grammar basics, parts of speech, stop words, morphology, dependencies, and applications like question answering and analyzing reviews.

Master natural language processing through parsing techniques, including part-of-speech tagging, constituency parsing, and dependencies, illustrated with context-free grammar and examples like the quick brown fox and cricket world cup.

Identify parts of speech such as nouns, verbs, adjectives, adverbs, pronouns, determiners, and practice syntactic analysis to determine word roles in sentences.

Explore lexicon-based tagging, rule-based tagging, and probabilistic data approaches for word tagging, with training data examples and 90 percent accuracy, plus deep learning with recurrent neural networks for sequential tagging.

Apply lexical tagging and simple statistics to assign the most frequent tags from the training corpus, then examine rule-based tagging and corpus analysis.

Create a lexicon and rule-based models, test with a 70/30 training-test split, compare unigram and bigram approaches, and use regex-backed backoff to improve tagging accuracy.

Explore stochastic parsing with hidden Markov models to tag word sequences using Markov assumptions, chain rule, and emission and transition probabilities, illustrated with a three-word example.

Apply the Viterbi algorithm to maximize the joint probability of word-tag sequences under a Markov assumption, using training data to compute initial and transition probabilities for accurate tagging.

Explore the theory of hidden Markov models and Markov chains, detailing transition and emission probabilities, hidden states, and how observations reveal sequential data.

Learn how to solve the decoding problem: given a word sequence and an HMM with initial, transition, and emission probabilities, find the most likely state sequence with the Viterbi algorithm.

Explore dynamic programming with the Viterbi algorithm to decode word sequences using start, transition, and emission probabilities learned from training data, including determinant, noun, and adjective.

Learn to estimate emission and transition probabilities for a hidden Markov model from training data, using simple count-based ratios to tag words and sequences.

Develop a Python notebook model using treebank data to train, compute emission and transition probabilities, and build a word-tag probability matrix with visualization via a heat map.

Demonstrates building a sequence model for syntactic processing, computing emission and transition probabilities, decoding the most probable state, and evaluating accuracy to guide improvements.

Explore recurrent neural networks (RNNs) and their role in advancing sequential models beyond traditional hidden Markov models, with insights into neural networks and deep learning.

Explore core parsing techniques in natural language processing, including constituency and dependency parsing, context-free grammars, bottom-up and top-down algorithms, and probabilistic CFG, to analyze complex sentences.

Examine why shallow parsing misses grammar and dependencies, and learn deeper techniques like constituency parsing to reveal word roles and relations, illustrated by man bites dog versus dog bites man.

We identify grammatical patterns by dividing sentences into constituents, define noun phrases, verb phrases, and prepositional phrases, and show how combining these can yield syntactically correct sentences via context-free grammars.

Explore context-free grammar through production rules that define how words form noun phrases and other constituents, using terminals, non-terminals, and parse trees to parse sentences.

Starting from the sentence symbol, explore top-down parsing to build a parse tree using grammar rules like np and vp, with backtracking to handle dead ends and multiple solutions.

Master top-down parsing with a CFD grammar and a recursive descent approach to analyze a complex sentence, visualize parse trees, backtrack, and edit grammar in a CFD editor.

Explore Kalidas bottom up passing, starting from words and building grammar driven reductions via a shift-reduce parser, with a stack based derivation to S.

Explore a bottom-up parsing case study using shift-reduce with a given grammar to parse strings, visualize the parse, and experiment with grammar edits, addressing ambiguity through probabilistic context grammars.

Explore practical issues in disambiguating meaning in NLP using a fish-and-net example, comparing bottom-up and top-down approaches, and applying a probabilistic method to resolve ambiguity.

The lecture demonstrates how probabilistic CFG resolves syntactic ambiguities by evaluating multiple parses in a bottom-up approach to distinguish meanings like look at the man with one eye.

Explore probabilistic context-free grammars that attach probabilities to production rules and compute parse likelihoods by multiplying branch probabilities, identifying the highest probability interpretation (e.g., astronomers saw stars with years).

Develop practical skills in building probabilistic grammars for language processing by implementing a case study that computes transition and emission probabilities, evaluates trees, and explores training concepts in Python.

Learn to convert a context-free grammar to cnf by breaking right-hand sides with new nonterminals, using A → BC or A → a, and handling ε for the start symbol.

Learn dependency parsing and dependency grammar, mapping how subjects, verbs, and objects relate across fixed and free word orders, with English and Hindi examples, and upcoming universal dependencies.

Explore dependency parsing in depth, identifying nominal subjects, verbs, and objects, and applying universal dependencies to parse sentence structure with root and modifiers.

Learn to build information extraction systems that turn unstructured text into structured data via named entity recognition, using rule-based and probabilistic methods with an airline data case study.

Build a flight information extraction system from a case study by labeling source city, destination, and time of day, using a dictionary and train/validation/test data.

Examine mapping words to indices and labels to indices, reverse dictionaries, and use IOB tagging to delineate inside, outside, and beginning of entities in NLP pipelines.

Analyze information extraction from unstructured text by outlining an NLP pipeline: preprocessing, sentence segmentation, sentence tokenization, part-of-speech tagging, entity and relation recognition, and record linkage.

Learn to build a sequence labeling pipeline that assigns IOB named-entity tags to tokens, using rule-based patterns, data formatting, and introductory probabilistic models.

Build unigram and bigram probabilistic taggers to assign the most likely label to each word, compare with rule-based methods, and apply to city, state, and country recognition using a dataset.

Develop a machine learning model for sequence classification to predict city, state, or country names using word features, gazetteers, and bigrams, with enhanced classifier performance.

Explore semantic processing to infer meaning from text, tackle word sense disambiguation and semantic association, and learn techniques like the Lesk algorithm to analyze semantics.

explores semantic processing and the meaning of text, showing how context disambiguates word senses and how words are represented as vectors linked to concepts; dumps act as the handle.

Explore how entities, entity types, and concepts create a meaning representation, using predicates to model associations and relationship types between instances and types.

This lecture explains arity in predicate logic, highlighting higher order predicates and the difficulty of verifying multiple true statements, and presents replication by breaking into smaller predicates.

Explore reification as a method to reduce higher-order semantics to lower-order predicates by creating virtual entities that unify multiple types, with seating roster and schema.org as illustrations.

Explore how schema dot org provides a semantic structure to specify hotel entities, ratings, prices, and attributes. See how search engines use these mappings to index pages and distinguish terms.

Explore semantic associations like is-a and is-in relationships, with examples such as a cat is a mammal and a wallet in a car, and discuss how instances and properties relate.

Explore how semantic associations and topic modeling identify the main topic of text by mapping relationships between concepts, from football and hockey to diabetes and universities.

Explore key nlp word relationships by examining hypernyms and hyponyms, antonyms, synonyms, meronyms and holonyms, and polysemy and homonyms through practical examples.

Explore the principle of composition and how combined words retain or alter meaning, with examples like red bull, heavy duty, and raining cats and dogs, to inform semantic algorithms.

Explore WordNet, a semantic lexical database for English useful for semantic processing, that groups words into cognitive synonyms called Seznec and links them through hypernym and hyponym relations.

Identify the correct sense of ambiguous words through supervised and unsupervised word sense disambiguation, using posterior and log probabilities with training data and the Lisk algorithm.

Explore unsupervised learning with the LEKS algorithm, implementing word sense disambiguation in Python using WordNet definitions, tokenization, and stopword removal.