Hands On Natural Language Processing (NLP) using Python

Install the Anaconda Python 3.6 distribution to access essential NLP packages like sklearn and nltk, and set up your environment across Windows, Mac, or Linux with the graphical installer.

Explore the Spyder IDE for Python coding, including opening Spyder from Anaconda, using the IPython console, setting the working directory, creating temp.py, and running code with run and shift+enter.

Learn to print in Python, work with variables, basic operations, and comments; understand dynamic typing, string repetition, and controlling newlines with end and \n.

Explore Python conditional statements guided by indentation, covering if and else blocks. Use examples of grade thresholds and divisibility checks with the modulus operator and printing results.

Explore Python loops, including while and for loops, with practical examples using range, indentation, and nested loops to print patterns and control line endings.

Explore Python loop control statements: break exits loops, continue skips iterations, and pass acts as a placeholder, demonstrated through simple loops and conditional logic.

Explore Python lists as the first data structure, learning non-homogeneous elements, indexing, printing, and list operations like append, insert, update, delete, and iteration.

Explore Python tuples as immutable, read-only alternatives to lists, learn how to declare single-element tuples with a trailing comma, access elements, concatenate, measure length, and iterate.

Explore Python dictionaries as key-value stores, declare and populate them with curly braces, access elements by keys, use get with defaults, delete with del, and iterate via keys() and values().

Learn console input and output in Python with input, casting to numbers, and displaying results; then write, append, and read files using with open and modes w, a, and r.

Learn to define and call your own Python functions using def, pass arguments, return values, and create a custom length function with a for loop.

Master object oriented programming in python by learning classes and objects, defining methods with self, and creating and using constructors to build and manipulate objects.

Learn Python list and dictionary comprehensions, and generator comprehensions, to create and filter data in a single line, with examples from numbers and joining words into sentences.

Explore regular expressions in Python, learn how a regex defines a search pattern to match or substitute text, and use re.sub to replace digits for NLP preprocessing.

Learn regex in Python with the re library, using dot star, dot, and plus to match text patterns, including word characters versus any characters.

Learn to use match and search to perform local versus global pattern matching in Python, and verify starts with and ends with regex patterns using carrot and dollar.

Learn to substitute text in Python with sub, performing global or limited replacements, using flags for ignore-case matches and count to constrain substitutions, illustrated by Avengers to Justice League.

Explore shorthand character classes in Python regular expressions, including digits, word and non-word characters, and whitespace; learn to group characters, escape dots, and substitute with re.sub.

Learn to clean and normalize a list of sentences using Python regular expressions for preprocessing in natural language processing, removing non-word characters, digits, and extra spaces.

Explore numpy and pandas for data science with Python, mastering array creation, reshaping, slicing, and vectorized calculations to support NLP tasks like tf-idf and bag of words.

Learn pandas essentials, including series and data frames, perform operations on series, apply functions, convert types, use copy versus assignment, and load datasets with read_csv such as matches.csv.

Install the nltc library in Python and install its dependencies, then import nltc and run nltc.download to fetch components for NLP modeling.

Tokenize a paragraph into sentences with sentence tokenize and extract words with word tokenize in Python, using an example paragraph from Leonardo DiCaprio's Oscar winning speech.

Explore how stemming and lemmatization reduce words to base forms for feature extraction in NLP, noting that stemming may produce meaningless roots while lemmatization preserves meaning with forms like intelligent.

This lecture demonstrates stemming in Python using NLTK's Porter Stemmer, tokenizing a paragraph into sentences, applying stemming to each word, and reconstructing sentences from stemmed words.

Explore lemmatization with a word net lemmatizer, compare it to stemming, and see tokenization, word-level lemmatization, and use cases in text classification and chatbots.

Learn to remove stopwords with NLTK's stopwords, tokenize sentences and words, filter out common words, and rebuild sentences to improve feature extraction for sentiment analysis and text classification.

Learn how to tag words with their parts of speech using nltk's pos_tag, tokenize a paragraph, and generate a tagged paragraph where each word carries its part of speech.

Learn how to perform named entity recognition in Python with NLTK, including tokenization, POS tagging, and chunking to extract organizations, persons, and locations.

We build a bag of words model from three sentences by preprocessing to lowercase, tokenizing, and turning word presence into a binary document-term matrix.

Build a bag-of-words model from scratch by tokenizing the paragraph into sentences, then convert all characters to lowercase, remove punctuation, and normalize spaces with regex.

Create a word count dictionary mapping each unique word to its occurrences across the corpus, building the histogram for the bag-of-words model.

Filter the bag of words by selecting the top most frequent words using word counts and the heapq library, to build a practical vocabulary for text classification.

Build a bag of words model from 100 frequent words, encoding each document as a binary vector, then convert to a 2D numpy array for NLP processing.

Identify limitations of the bag-of-words model and learn how tf-idf improves text representations by combining term frequency with inverse document frequency, with Python from-scratch guidance.

build a tf-idf model from scratch by creating the idf matrix and mapping words to document counts and IDs, with next steps to build the tf matrix.

Build the tf matrix and compute tf-idf values by multiplying term frequency with inverse document frequency. Use a dataset of 21 sentences and 100 frequent words in Python.

Compute tf-idf scores by multiplying term frequencies with inverse document frequencies, building a tf-idf matrix across all words and documents.

Convert a tf-idf matrix in Python to a two-dimensional array, transpose it to 21 by 100, and align it with the bag-of-words concept for efficient text classification.

Explore n-gram modeling with markov chains to predict next characters or words, and build a character and word n-gram based autocomplete system.

Build a three-gram character model from a text, map each trigram to following characters, and generate autocomplete text by randomly selecting the next character.

Learn to build a word n-gram model in Python by tokenizing text into words, creating a trigram dictionary, and generating text.

Learn latent semantic analysis and singular value decomposition to map documents and terms into concepts, with tf-idf foundations and probabilities. See how this underpins article bucketing and search engine indexing.

Explore latent semantic analysis in Python by building concepts from seven sentences, transforming text with tf-idf vectorization and truncated SVD to reveal top keywords per concept for document classification.

Explore latent semantic analysis in Python, building a tf-idf model and concept-based document classification, with concept keywords, sentence scoring, and concept-word mappings.

Learn to use WordNet in nltk to find synonyms and antonyms by iterating synsets and lemmas, building a Python program that collects and deduplicates results.

Track negation in sentences by tokenizing with nltk, detecting not, and merging not with the following word to form not_ prefixed tokens like not_happy for improved NLP models.

Track negation words in python by replacing not followed by an adjective with its antonym using wordnet, improving nlp for sentences like I was not happy with the team's performance.

Fetch data from an external source, preprocess it, build an intermediate representation, and train a text classifier to predict positive or negative sentiment using the Cornell sentiment dataset.

Load a text dataset for sentiment classification by importing files from the txt_sent_token folder with sklearn's load_files, producing negative (0) and positive (1) classes and the X and Y data.

Persist the dataset by saving X and Y as pickle files to speed up loading on large datasets like 50,000 IMDb reviews.

Preprocess the dataset to build a sentiment classifier by creating a corpus of reviews. Apply regex to remove non-word characters, lowercase the text, and remove single characters.

Build a 2000-feature bag-of-words model with count vectorizer parameters (min_df 3, max_df 0.6, English stopwords) and transform the corpus into a 2000×2000 array, preparing for a tf-idf conversion.

Transform a simple bag-of-words model into a tf-idf representation using sklearn's TfidfTransformer, building on a pre-built CountVectorizer bag-of-words.

Split the 2000 reviews into 1600 training and 400 testing sets using sklearn's train_test_split, with tf-idf features and a fixed random state, preparing for logistic regression.

Explore logistic regression as a binary classifier for sentiment analysis, using tf-idf features and a 0.5 threshold to predict positive or negative documents, while learning optimal coefficients.

Import sklearn's logistic regression class, instantiate a classifier, and fit it on text_train to train the model. Preview test-set predictions to measure accuracy in the next video.

Learn to evaluate a trained logistic regression nlp classifier in Python using a confusion matrix, achieving about 84.75% accuracy on 400 test documents.

Persist the trained sentiment classifier by saving it as a pickle file and store the tf-idf vectorizer for real-time tweet sentiment predictions.

Import and load the pickled classifier and tf-idf vectorizer, transform a sample sentence, and predict its sentiment as positive or negative. Next, build a Twitter sentiment bot.

Set up a Twitter application to fetch real-time tweets and apply your logistic regression sentiment classifier, including creating API keys and tokens for authentication in a Python workflow.

Set up Tweepy authentication with consumer keys and access tokens, load the classifier and tf-idf models, and prepare the Twitter sentiment analysis workflow in Python.

Set up oauth authentication for a Twitter client with consumer key and secret and access token and secret, then configure Tweepy to fetch tweets by a search term.

Fetch the top 100 recent tweets about Facebook using Tweepy's search cursor, filtering retweets and English language. Prepare tweet texts for future sentiment analysis with tf-idf and a classifier.

Load the tf-idf model and vectorizer from pickle files, load the classifier, and test a sample tweet to classify it into a category.

Preprocess tweets by removing links and symbols, lowercase the text, expand contractions, and normalize punctuation and digits. Feed the cleaned text into the vectorizer and classifier to forecast sentiment.

Apply a vectorizer and classifier in Python to predict and print each tweet's sentiment as positive or negative, then plot the recent 100 tweets for a Facebook query.

Count positive and negative tweets from predicted sentiments and plot a two-bar chart with matplotlib and numpy, labeling axes and visualizing the results.

Build a simple natural language processing text summarizer that fetches Wikipedia articles, tokenizes and preprocesses text, builds a weighted histogram with bag-of-words and tf-idf, and selects top sentences.

Fetch data from Wikipedia using Python by scraping a Wikipedia page with BeautifulSoup and lxml to extract the page text.

Parse the Wikipedia global warming article with Beautiful Soup to extract paragraph text from p tags, then remove references in square brackets with a regular expression and clean extra spaces.

Learn data preprocessing in Python by lowercasing text, removing non-word characters, digits, and extra spaces, producing a clean text for histogram analysis and a separate main text for the summary.

Tokenize the paragraph into sentences using nltk, then prepare English stopwords to exclude common words when creating a normal and weighted word histogram.

Tokenize and clean the paragraph, build a word-to-count histogram in a dictionary excluding stopwords, then derive a weighted histogram by normalizing counts to the maximum.

Create a sentence scoring dictionary by iterating over sentences, tokenizing, weighting words, and updating scores; apply a length filter to keep sentences under 30 words.

Learn to generate a text summary by ranking sentences with a sentence score dictionary and selecting the top n using heapq, demonstrated on a Wikipedia article about global warming.

This lecture introduces the word two vec model, showing how word vectors capture semantic relations and enable analogies like king minus man plus woman, trained on large corpora.

Fetch a single Wikipedia article on global warming, preprocess the text, and build a vertovec model using gensim, after installing BeautifulSoup and lxml.

Fetch and pre-process Wikipedia data on global warming, remove punctuation, and tokenize the text. Transform into tokenized sentences and train a Word2Vec model with gensim.

Train a Word2Vec model in Python using a preprocessed list of sentences from the global warming article, set min_count, and explore the resulting vocabulary and word similarities.

Explore extracting word vectors with a trained word2vec model, inspecting a 100‑dimensional vector for global, and finding most similar words to warming, with vector arithmetic and stop-word concerns.

this lecture revisits refining a natural language processing model in python by correctly handling punctuation and symbols, removing stopwords, training word vectors, and evaluating word similarity to improve efficiency.

Explore pre-trained models like google news vectors -300 with word2vec in python using gensim, and learn for similarity, vector arithmetic, and memory considerations.