U&P AI - Natural Language Processing (NLP) with Python

Explore fundamental and advanced feature engineering methods for text data. Learn how turning unstructured text into numerical features powers NLP models, with notes on advantages, disadvantages, and examples.

Explore the count vectorizer to convert text into features by counting word occurrences, addressing frequency limitations of binary bag-of-words representations.

This lecture explains n-grams as a solution to losing context when analyzing text, showing unigram, bigram, and trigram features and using a count vectorizer to generate them with Python libraries.

Learn hash vectorizing in natural language processing with Python, using the hashing trick to encode text as memory efficient numerical indices. Produce fixed-size vectors for supervised or unsupervised tasks.

Word embedding maps words to vectors that capture context and semantic relations. CBOW and skip-gram models trained on corpora learn word similarities and analogies, such as king to queen.

Download external, labeled movie review corpus using the categorized plaintext corpus reader, then access positive and negative categories and print sample documents for sentiment analysis.

Explore frequency distribution on informal web text corpora, counting distinct words and revealing the ten most common tokens, while learning how preprocessing improves results in NLP with Python.

Explore how WordNet stores word senses in a hierarchical database and how to retrieve definitions, synonyms, and example sentences for words like chair and bat, using NLTK's WordNet.

Explore WordNet basics by extracting hyponyms and hypernyms in Python, tracing word senses from root to leaf nodes, and identifying direct parent terms like adult and female for woman.

Learn how to compute the average policy for nouns using WordNet senses and part of speech filters, with hands-on Python code to gather distinct lemmas.

Explore tokenization by segmenting text into tokens and building a lexicon for NLP. Convert these tokens into bag-of-words vectors to power machine learning, document retrieval, and search.

Explore tokenizing sentences with whitespace using the split method, address punctuation in tokens, and convert words to numerical vector representations for NLP tasks.

Build a vocabulary of unique tokens, encode documents with one-hot vectors, and explore vocabulary size using Banda's data frames with a default integer row index.

Convert sentences into one-hot and bag-of-words vectors to preserve meaning for neural language models and NLP pipelines, enabling efficient, scalable processing of large vocabularies.

Convert text into a binary vector that marks word presence in a sentence. Use a dictionary-based sparse representation for vocabulary to store only present words.

Understand the dot product, aka inner or scalar product, which yields a single value from elementwise vector multiplication and summation, and its link to matrix product and inner joins.

Use bag-of-words and dot product to measure sentence similarity, explore vector space models, and examine binary vector representations for efficient document retrieval and text search.

Explore tokenization that splits on whitespace and punctuation, decide whether punctuation becomes tokens or is ignored, and use regex to group words, reducing vocabulary dimensionality in NLP.

Analyze the role of stop words in NLP, balancing vocabulary size, computational efficiency, and information content, and learn when to apply domain-specific filters without losing crucial two-grams.

Normalize capitalization to reduce vocabulary size and memory usage, consolidate similar spellings, and strengthen cross-token meaning, while balancing potential loss of information and preserving proper nouns for named entity recognition.

Compare stemming and lemmatization by tradeoffs in recall and precision, noting that stemmers are faster but increase ambiguity, while lemmatization preserves meaning for more accurate search results.

Explore how sentiment analysis measures positivity in text using NLP pipelines and outputs a score between minus one and plus one. Compare rule-based and machine learning approaches.

Explore two sentiment analysis approaches: a rule-based heuristics method using a keyword dictionary and token processing, and a data-driven machine learning approach trained on labeled text to predict sentiment scores.

Learn sentiment analysis in Python by building and evaluating a name-based baseline model, incorporating tokens, negation handling with Vader, and improving accuracy through data-frame driven experiments.

Explore turning word counts into continuous representations for NLP, using bag-of-words, frequency-based methods, and tf-idf with n-grams to improve keyword search, spam filters, and sentiment analysis.

Examine the bag-of-words approach by counting word occurrences, normalizing term frequency, and filtering stop words through tokenization to compare documents in natural language processing with Python.

Turn text into vectors by counting tokens, building a shared lexicon across documents, normalizing counts, and creating fixed-length document vectors for analysis.

Explore the vector space representation of documents and the corpus vocabulary, then use cosine similarity and dot products to compare document vectors.

Learn how term frequency and inverse document frequency combine to form tf-idf, weighting words by their importance across a corpus to enable topic modeling and document comparison.

Replace word counts with tf-idf to create meaningful document vectors, use cosine similarity for retrieval, and convert the tf-idf index into a chat system by pairing questions with responses.

Explore how tf-idf weights words by inverse document frequency to build vector space representations of word meanings, enabling cosine distance as a primary measure of semantic similarity.

Explore how keyword search compares with semantic search by building topic vectors from tf-idf scores and latent semantic analysis to represent meaning and improve document retrieval.

Address tf-idf limitations for semantic search by introducing topic vectors for words and documents, handling synonyms and polysemy, and reducing dimensions for meaningful semantics.

Transform tf-idf word frequencies into topic vectors by weighting words for three topics, using a 3xN weight matrix and matrix multiplication to obtain document topic representations.