Natural Language Processing (NLP) with NLTK and Scikit-learn

This video gives an overview of the entire course.

In this video, we use the Python NLTK library to understand more about the POS tagging features in a given text.

• Create a variable called simpleSentence

• Invoke the NLTK built-in tokenizer function word_tokenize()

• Invoke the NLTK built-in tagger pos_tag()

Now, we will explore the NLTK library by writing our own taggers. We’ll write various types of taggers such as Default tagger, Regular expression tagger and Lookup tagger.

• Define a new Python function called learnDefaultTagger

• Create an object of the DefaultTagger() class

• Call the tag() function of the tagger object

Next, let’s learn how to train our own tagger and save the trained model to disk so that we can use it later for further computations.

• Define a function called sampleData()

• Define a function called buildDictionary()

• Build an nltk.UnigramTagger() object

This video will teach us how to define grammar and understand production rules.

• Import the generate function from the nltk.parse.generate

• Define a new grammar

• Create a new grammar object using the nltk.CFG.fromstring()

Probabilistic CFG is a special type of CFG in which the sum of all the probabilities for the non-terminal tokens (left-hand side) should be equal to one. Let's write a simple example to understand more.

• Identify tokens in the grammar

• Join the list of all the production rules into a string

Recursive CFGs are a special types of CFG where the Tokens on the left-hand side are present on the right-hand side of a production rule. Palindromes are the best examples of recursive CFG.

• Create a new list data structure called productions

• Add production rules that define palindromes

• Pass the newly constructed grammarString to the NLTK built-in nltk.CFG.fromstring function

In this video, we will learn how to use the in-built chunker. We will use some features that will be used from NLTK as part of this process.

• Add string to a variable called text

• Break the given text into multiple sentences

• Do POS analysis using the default tagger

Now that we know using the built-in chunker, in this video, we will write our own Regex chunker.

• Write regular expressions

• Understand tag patterns

• Identify chunks

In this video, we will learn the training process, training our own chunker, and evaluating it.

• Import the conll2000 corpus and treebank corpus

• Define a new function, mySimpleChunker()

• Create a list of two datasets

Recursive descent parsers belong to the family of parsers that read the input from left to right and build the parse tree in a top-down fashion and traversing nodes in a pre-order fashion.

• Define a new function, RDParserExample

• Iterate over the list of sentences in the textlist variable

• Create a new CFG object using grammar

In this video, we will learn to use and understand shift-reduce parsing.

• Define a new function, SRParserExample

• Iterate over the list of sentences in the textlist variable

• Define two sample sentences to understand the shift-reduce parser

We will now learn how to parse dependency grammar and use it with the projective dependency parser.

• Create a grammar object using the nltk.grammar.DependencyGrammar class

• Define the sample sentence on which parser will be run

Chart parsers are special types of parsers which are suitable for natural languages as they have ambiguous grammars. Let’s learn about them in detail.

• Import CFG module, ChartParser and BU_LC_STRATEGY features

• Create a sample grammar for the example

• Acquire all the parse trees

Python NLTK has built-in support for Named Entity Recognition (NER). Let’s learn to use inbuilt NERs.

• Define a new function called sampleNE()

• Define a function called sampleNE2()

• Call the two sample functions

Is it possible to print the list of all the words in the sentence that are nouns? Yes, for this, we will learn how to use a Python dictionary.

• Define a new class called LearningDictionary

• Create buildDictionary() and buildReverseDictionary()

• Define getPOSForWord()

Choosing the feature set Features are one of the most powerful components of nltk library. They represent clues within the language for easy tagging of the data that we are dealing with.

• Create learnSimpleFeatures()

• Create learnFeatures()

• Compare both the functions

A natural language that supports question marks (?), full stops (.), and exclamations (!) poses a challenge to us in identifying whether a statement has ended or it still continues after the punctuation characters. Let’s try and solve this classic problem.

• Define featureExtractor()

• Create segmentTextAndPrintSentences()

• Extract all the features from the traindata and store it in traindataset

In previous videos, we have written regular-expression-based POS taggers that leverage word suffixes, let’s try to write a program that leverages the feature extraction concept to find the POS of the words in the sentence.

• Indicate the dual behavior of the words

• Define a new function called withContextTagger()

• Build a featuredata list

In computing, a pipeline can be thought of as a multi-phase data flow system where the output from one component is fed to the input of another component.

• Create new empty list to keep track of all the threads in the program

• Define a new function, extractWords()

The text similarity problem deals with the challenge of finding how close given text documents are.

• Define an IDF that finds the IDF value

• Define a TF_IDF

• Display the contents of vectors

In many natural languages, while forming sentences, we avoid the repeated use of certain nouns with pronouns to simplify the sentence construction.

• Define a new class called AnaphoraExample

• Create a unique list of males and females

• Create a NaiveBayesClassifier object called _classifier

In previous videos, we learned how to identify POS of the words, find named entities, and so on. Just like a word in English behaves as both a noun and a verb, finding the sense in which a word is used is very difficult for computer programs.

• Define a function with the name understandWordSenseExamples()

• Define a new function, understandBuiltinWSD()

• Define a new variable called maps

Feedback is one of the most powerful measures for understanding relationships. In order to write computer programs that can measure and find the emotional quotient, we should have some good understanding of the ways these emotions are expressed in these natural languages.

• Define a new function, wordBasedSentiment()

• Define sample text to analyze

• Create multiWordBasedSentiment()

Let’s write our own sentiment analysis program based on what we have learned in the previous video.

• Define a new function, mySentimentAnalyzer()

• Extract the sentences from the variable feedback

Conversational assistants or chatbots are not very new. One of the foremost of this kind is ELIZA, which was created in the early 1960s and is worth exploring. NLTK has a module, nltk.chat, which simplifies building these engines by providing a generic framework. Let’s see that in detail.

• Define builtinEngines()

• Create a new function called myEngine()

• Define a nested tuple data structure