Natural Language Processing with Python: 3-in-1

Our first task involves us learning how to access any one of these corpora. We have decided to do some tests on the Reuters corpus or the same.

Now that we have learned how to load and access an inbuilt corpus, we will learn how to download and also how to load and access any external corpus.

The objective of this video is to get you to perform a simple counting task on any given corpus.

Web and chat text corpus is non-formal literature that, as the name implies, contains content from Firefox discussion forums, scripts of movies, wine reviews, personal advertisements, and overheard conversations. Our objective here in this video is to understand the use of frequency distribution and its features/functions.

From this video onwards, we will turn our attention to WordNet. As you can read in the title, we are going to explore what word sense is.

A hyponym is a word of a more specific meaning than a more generic word such as bat, which we explored in the introduction section of our previous video.

This video is different from previous videos. It's not just an API concept discovery but we are going to discover a linguistic concept here.

As an NLP expert, you are going to work on a lot of textual content. And when you are working with text, you must know string operations. We are going to start with a couple of short and crisp recipes that will help you understand the str class and operations with it in Python.

Moving ahead from the previous video, we will see substrings, string replacements, and how to access all the characters of a string.

We start off with a small video for accessing PDF files from Python. For this, you need to install the PyPDF2 library.

In this video, we will see how to load and read Word/DOCX documents. The libraries available for reading DOCX word documents are more comprehensive, in that we can also see paragraph boundaries, text styles, and do what are called runs.

For this video, we are not going to use anything new in terms of libraries or concepts. We are reinvoking the concept of corpus from the first section. Just that we are now going to create our own corpus here instead of using what we got from the Internet.

The objective of this video is to read such an RSS feed and access content of one of the posts from that feed. For this purpose, we will be using the RSS feed of Mashable.

Most of the times when you have to deal with data on the Web, it will be in the form of HTML pages. For this purpose, we thought it is necessary to introduce you to HTML parsing in Python.

Understand the meaning of tokenization, why we need it, and how to do it.

Let's understand the concept of a stem and the process of stemming. We will learn why we need to do it and how to perform it using inbuilt NLTK stemming classes.

Understand what lemma and lemmatization are. Learn how lemmatization differs from Stemming, why we need it, and how to perform it using nltk library's WordnetLemmatizer.

We will be using the Gutenberg corpus as an example in this recipe. The Gutenberg corpus is part of the NLTK data module. It contains a selection of 18 texts from some 25,000 electronic books from the project Gutenberg text archives.

Edit distance, also called as Levenshtein distance is a metric used to measure the similarity between two distances.

This video is supposed to give you an idea of how to handle a typical text analytics problem when you come across it.

We start off with a video that will elaborate the use of the, +, and ? operators in regular expressions.

The starts with (^) and ends with ($) operators are indicators used to match the given patterns at the start or end of an input text. Let’s study about them in detail.

In this video, we shall run some iterative functions with regular expressions. More specifically, we shall run multiple patterns on an input string with a “for” loop and we shall also run a single pattern for multiple matches on the input. Let's directly see how to do.

In this video, we shall first run a simple date regex. Along with that, we will learn the significance of the () groups. Since that's too less to include in a recipe, we shall also throw in some more things like the squared brackets [], which indicate a set.

We have covered all the important notations that I wanted to cover with examples in the previous videos. Now, going forward, we will look at topics that are geared more towards accomplishing a certain task using regular expressions than explaining any notations.

We already know the concepts of tokens, tokenizers, and why we need them from the previous section. We have also seen how to use the inbuilt tokenizers of the NLTK module. In this video, we will write our own tokenizer; it will evolve to mimic the behavior of nltk.word_tokenize().

We already know the concept of stems/lemmas, stemmer, and why we need them from the previous section. We have seen how to use the inbuilt porter stemmer and Lancaster stemmer of the NLTK module.

In this video, we use the Python NLTK library to understand more about the POS tagging features in a given text.

Now, we will explore the NLTK library by writing our own taggers. We’ll write various types of taggers such as Default tagger, Regular expression tagger and Lookup tagger.

Next, let’s learn how to train our own tagger and save the trained model to disk so that we can use it later for further computations.

This video will teach us how to define grammar and understand production rules.

Probabilistic CFG is a special type of CFG in which the sum of all the probabilities for the non-terminal tokens (left-hand side) should be equal to one. Let's write a simple example to understand more.

Recursive CFGs are a special types of CFG where the Tokens on the left-hand side are present on the right-hand side of a production rule. Palindromes are the best examples of recursive CFG.

In this video, we will learn how to use the in-built chunker. We will use some features that will be used from NLTK as part of this process.

Now that we know using the built-in chunker, in this video, we will write our own Regex chunker.

In this video, we will learn the training process, training our own chunker, and evaluating it.

Recursive descent parsers belong to the family of parsers that read the input from left to right and build the parse tree in a top-down fashion and traversing nodes in a pre-order fashion.

In this video, we will learn to use and understand shift-reduce parsing.

We will now learn how to parse dependency grammar and use it with the projective dependency parser.

Chart parsers are special types of parsers which are suitable for natural languages as they have ambiguous grammars. Let’s learn about them in detail.

Python NLTK has built-in support for Named Entity Recognition (NER). Let’s learn to use inbuilt NERs.

Is it possible to print the list of all the words in the sentence that are nouns? Yes, for this, we will learn how to use a Python dictionary.

Choosing the feature set Features are one of the most powerful components of nltk library. They represent clues within the language for easy tagging of the data that we are dealing with.

A natural language that supports question marks (?), full stops (.), and exclamations (!) poses a challenge to us in identifying whether a statement has ended or it still continues after the punctuation characters. Let’s try and solve this classic problem.

In previous videos, we have written regular-expression-based POS taggers that leverage word suffixes, let’s try to write a program that leverages the feature extraction concept to find the POS of the words in the sentence.

In computing, a pipeline can be thought of as a multi-phase data flow system where the output from one component is fed to the input of another component.

The text similarity problem deals with the challenge of finding how close given text documents are.

In many natural languages, while forming sentences, we avoid the repeated use of certain nouns with pronouns to simplify the sentence construction.

In previous videos, we learned how to identify POS of the words, find named entities, and so on. Just like a word in English behaves as both a noun and a verb, finding the sense in which a word is used is very difficult for computer programs.

Feedback is one of the most powerful measures for understanding relationships. In order to write computer programs that can measure and find the emotional quotient, we should have some good understanding of the ways these emotions are expressed in these natural languages.

Let’s write our own sentiment analysis program based on what we have learned in the previous video.

Conversational assistants or chatbots are not very new. One of the foremost of this kind is ELIZA, which was created in the early 1960s and is worth exploring. NLTK has a module, nltk.chat, which simplifies building these engines by providing a generic framework. Let’s see that in detail.