U&P AI - Natural Language Processing (NLP) with Python

Explore fundamental and advanced feature engineering methods for text data. Learn how turning unstructured text into numerical features powers NLP models, with notes on advantages, disadvantages, and examples.

Explore the count vectorizer to convert text into features by counting word occurrences, addressing frequency limitations of binary bag-of-words representations.

This lecture explains n-grams as a solution to losing context when analyzing text, showing unigram, bigram, and trigram features and using a count vectorizer to generate them with Python libraries.

Learn hash vectorizing in natural language processing with Python, using the hashing trick to encode text as memory efficient numerical indices. Produce fixed-size vectors for supervised or unsupervised tasks.

Word embedding maps words to vectors that capture context and semantic relations. CBOW and skip-gram models trained on corpora learn word similarities and analogies, such as king to queen.

Download external, labeled movie review corpus using the categorized plaintext corpus reader, then access positive and negative categories and print sample documents for sentiment analysis.

Explore frequency distribution on informal web text corpora, counting distinct words and revealing the ten most common tokens, while learning how preprocessing improves results in NLP with Python.

Explore WordNet basics by extracting hyponyms and hypernyms in Python, tracing word senses from root to leaf nodes, and identifying direct parent terms like adult and female for woman.

Learn how to compute the average policy for nouns using WordNet senses and part of speech filters, with hands-on Python code to gather distinct lemmas.

Explore tokenization by segmenting text into tokens and building a lexicon for NLP. Convert these tokens into bag-of-words vectors to power machine learning, document retrieval, and search.

Explore tokenizing sentences with whitespace using the split method, address punctuation in tokens, and convert words to numerical vector representations for NLP tasks.

Build a vocabulary of unique tokens, encode documents with one-hot vectors, and explore vocabulary size using Banda's data frames with a default integer row index.

Convert sentences into one-hot and bag-of-words vectors to preserve meaning for neural language models and NLP pipelines, enabling efficient, scalable processing of large vocabularies.

Understand the dot product, aka inner or scalar product, which yields a single value from elementwise vector multiplication and summation, and its link to matrix product and inner joins.

Examine the bag-of-words approach by counting word occurrences, normalizing term frequency, and filtering stop words through tokenization to compare documents in natural language processing with Python.

Explore the vector space representation of documents and the corpus vocabulary, then use cosine similarity and dot products to compare document vectors.

Learn how term frequency and inverse document frequency combine to form tf-idf, weighting words by their importance across a corpus to enable topic modeling and document comparison.

Explore how tf-idf weights words by inverse document frequency to build vector space representations of word meanings, enabling cosine distance as a primary measure of semantic similarity.

Address tf-idf limitations for semantic search by introducing topic vectors for words and documents, handling synonyms and polysemy, and reducing dimensions for meaningful semantics.