Project based Text Mining in Python

In this lecture, the students will learn about the need of transforming textual data into a structured format so that they can be consumed by machine learning techniques for analysis. Textual data is raw and is far from being structured despite of having paragraphs and sentences. In terms of machine learning the data should be in the form of a matrix having rows and columns with each cell holding a numerical value. In this lecture we will look into structuring the textual data and applying different representation schemes on it.

In this lecture, we discuss the Bag-of-Words (BoW) approach while vectorizing our textual data. In doing so, the position relation information is lost about the words or tokens, however, their frequency related information is retained.

In this lecture, we discuss the most frequently used representation scheme for text that is term frequency - inverse document frequenty or tf-idf. It is very effective towards noise and domain specific stopwords.

In this lecture, we implemented the text representation technique with a very basic corpus consisting of only one document. The document is converted in a matrix of one row and few features representing unique words in the corpus.

In this lecture, a multi-document corpus is structured to generate a matrix representation with multiple row representing document. A cell value represents the frequency or binary value of the feature in column in the document in row.

In this video, we experiment with the different parameter settings for the constructor CountVectorizer. Setting the binary = 'true' converts all the values that are greater than 1 to 1. The parameters max_df and min_df define the maximum and minimum range of document frequencies for a word. Anything beyond this range is ignored. max_features help to restrict the number of features to a specific number. ngram_range helps to provide a lower and upper limit for ngrams to be considered as features.

This lecture covers representing the textual documents in a structured format while having tf-idf values for each word in a document. These values are generated with the formula. For any word w in document d,

TF-IDF(w) = TF(w) x IDF(w)

TF(w) = count of w in d / size of d

IDF(w) = log [M +1 / k]

Where M is the total number of documents while k is the count of documents containing w.

Applying the text representation techniques on a dummy dataset file by reading data file from the disk and organizing it into list of documents as strings. Separating the input documents from their labels and then transforming the input documents to a structured format.

Applying the text representation techniques on a dataset from UCI repository.

This lecture introduces the need for machine learning as a kind of inductive learning. It is a kind of learning in which the model learns from patterns in the input data. It is highly desired for automation as it reduces human effort and manual involvement by a great amount.

This lecture overviews the definitions of machine learning and explains them with the help of an example. For those who are new to machine learning, it will serve to lay the foundation based on which the coming concepts of machine learning can be grasped.

Machine learning is a multi-disciplinary area and therefore, needs a basic understanding of multiple subject areas to master it. Probability and statistics are of key importance among others. There is a thin boundary between data mining and machine learning. Generally, it is said that if only patterns are recognized from the data then it is data mining but if those patterns are used for automatic decision making, then it is machine learning. Neurocomputing is important in the sense that many well-performing models come from that domain. That's why most of us are aware of the word deep learning.

There are different types of machine learning techniques. However, the most basic ones are supervised and unsupervised learning. Supervised learning is learning with labels available and has two types i.e., classification and regression. Unsupervised learning is learning without labels and is called clustering. The other types of learning are reinforcement learning and recommender systems. Of course, there are many other types of techniques that arise from these basic forms of learning e.g., semi-supervised learning, online learning, multi-task learning, meta-learning, etc.

In this lecture, we discuss pattern recognition which lies at the heart of machine learning with the help of few examples.

This lecture gives elaborates on the end to end pipeline of any machine learning project. What are the different components and how they collaborate to provide a working model. Since, we are also going to apply machine learning techniques on textual data, therefore, it is very important for us to be aware of the challenges and advantages of using ML approaches. For example, later in this module, we will be using classification algorithms which are supervised machine learning approaches. The pipeline helps us to be aware of issues that may arise at different stages of a  project and how address them.

This lecture gives an overview of machine learning. In order to classify documents we will be using the machine learning classifiers on textual data. Classification is supervised learning where the training data are supplied with labels.

In this lecture, students will learn about KNN classifier, and its working mechanism. We also discuss different distance measures used with KNN and provide commentary on the use of cosine similarity over other distance measures.

In this lecture, we discuss the Naive Bayes classifier. The unseen document has its probability calculated with all the possible classes and the one with highest probability is decided as the class for the unseen document.

In this lecture we have studied how different classifiers can be applied to structured textual documents to train a model. Next we saw how a trained model can be used to predict the label for an unseen document and check how confident the model is in predicting such a label.

In this lecture we modify our classifiers from the default settings with different parameter values and discuss that their impact will be towards the training and prediction of the model.

Correction in video: the boolean parameter fit_prior = True (i.e., boolean true) and not 'true' as string.

In this lecture we have acquired a labeled dataset from the UCI repository and have trained our model on it and  while keeping the last few documents to test how good the model can predict the label for an unseen document.

In this lecture, we introduce the theoretical concepts of clustering with the help of examples. It explains the two types of clustering approaches, which are partitional clustering and hierarchical clustering. An overview of the working of these approaches is presented.

In this lecture we discuss the working of the K-Means Clustering algorithm as a kind of partitional clustering that split the data into K clusters and then keep on switching objects or documents among clusters until the best separation is achieved.

The lecture demonstrates the use of KMeans class from sklearn library to separate the documents of a small dataset into multiple clusters using Kmeans algorithm.

This lecture introduces the nearest neighbors clustering algorithm. Its a radius or threshold based algorithm that requires the value of threshold and identifies the desired number of clusters and the points that belong to them by itself.

The lecture demonstrate the use of NearestNeighbors class from sklearn to apply on a dataset and find its nearest neighbors either based on the number of neighbors or the radius, each with a reference point.

This lecture elaborates the background working of hierarchical clustering. It discusses agglomerative and divisive hierarchical clustering approaches with the help of an example.

In this lecture we implement Agglomerative hierarchical clustering with default settings.

This lecture explains the concept of linkage for calculating the distance between an object and a cluster or two clusters. There are four different types of linkage measures they are single linkage, complete linkage, average linkage and wards linkage.

In this lecture we modify the default nature of agglomerative clustering provide different parameters to it.

In this lecture we perform  clustering on a dataset from UCI repository consisting of raw text. The main issue is to deal with real datasets while making sure that are getting the required data in correct format.

In this lecture we discuss the parameters of the clustering objects that are provided in the constructor. Using these parameters we can modify the behavior of the algorithm. The effect of some parameters is be observed in the final output while others are related to the internal working of the algorithm.

This lecture elaborate on the use of sequared error for finding the suitable number of cluster for a given dataset.

In this lecture, we evaluate inertia or sum of squared error for a given number of clusters and evaluate what would be a suitable number of clusters for a given dataset.

In this lecture, we introduce the module that is about validation and evaluation techniques and their importance in verifying the performance of a model.

In this lecture we discuss the cross validation techniques for evaluating the performance of a model.

In this lecture we studied the implementation of validation, where the dataset is separated into training and test sets.

In this lecture we apply KFold cross validation to the data to separate our dataset into K sets and then use k-1 sets to train the model and 1 set to test the model. The process is repeated K times so that each set get a chance of being part of the test case.

In this lecture, we separate our dataset into training and testing sets using Leave one out validation approach, which is an extreme case of KFold cross validation with the value of K equal to the total number of instances.

In this lecture we discuss the evaluation techniques used for comparing different classifiers. It is important to know how good a model can predict before applying it to unseen data. Therefore, labels for some of the instances from the training data are hidden and are treated as test cases. Then the actual or true labels are compared with the predicted labels to know the predictive accuracy of a model.

In this lecture we apply text representation, classification, validation and evaluation techniques to implement a model, train it and then evaluate it to see how good it is trained.

In this lecture we calculate precision, recall and f1 score for a classifier to measure how good it has predicted the unknown labels

In this lecture, we evaluate confusion matrix for the labels of the test set to identify where the model is making most of its mistakes.

In this lecture we put all the pieces together to train multiple classifiers, use a validation scheme and evaluate these classifiers to see which has performed better.

In this lecture, we discuss the clustering evaluation techniques that are grouped as internal and external evaluation techniques.

In this lecture we study the text normalization schemes that help in converting textual data to its normal form by removing noise and substituting words by their basic representation.

In this lecture, we convert our dataset to a lower case and remove unwanted white spaces and punctuations

In this lecture we remove stopwords from our dataset using ENGLISH_STOP_WORDS list from the feature_extraction.stop_words package in sklearn library.

In this lecture, we will reduce words to their basic forms or stems to reduce the number of features.

In this lecture, we study how regular expressions work. They are going to help us in removing domain specific stopwords i.e., unwanted sequence of characters from source depending on from where we have acquired our dataset.

In this lecture we apply regular expressions on our dataset to remove the domain specific stopwords

In this lecture we will investigate the parts of speech for a word in a sentence. Parts of speech may help in a number of ways where

In this lecture we implement the data acquisition techniques for reading data from a webpage. Repeating the process for multiple pages can help prepare our own dataset.

In this lecture, we use the actual procedure for text segmentation into sentences and tokenization into words or tokens. So far we were using the basic string function of splitting it on various characters, however, they make mistakes by not considering various complicated cases.

In this lecture we discuss topic modeling as a dimensionality reduction and soft clustering technique.

In this lecture we discuss the plate notations used to diagrammatically represent topic models.

In this lecture we discuss the working mechanism of topic models i.e., Latent dirichlet allocation or LDA in particular. LDA has many variants with supervised, unsupervised, semi-supervised, transfer learning and knowledge based approaches and therefore, in order to understand them and realize which would cater best to a particular application need, it is very important to learn about its working or internal mechanism. When we use it through a library, we don't get to realize the internal working which is good enough in case we need to use its basic implementation. Most of the variants are at research level and therefore, are not introduced as library functions.

In this lecture, we implement a topic model (Latent dirichlet allocation) with the default settings and applied it on our dummy dataset. Next we structured the output to make it more meaningful.

In this lecture we apply topic modeling (LDA) on UCI repository dataset (Eco-hotel reviews) to separate all the reviews into 6 topics each representing a concept or subject discussed across multiple reviews.

In this lecture, the role of hyper-parameters in topic models is explained. These values are going to decide if we want more topics with reasonable representation within a document or fewer topics with higher representation. Similarly, if we want to have fewer words with high representation within a topic or more words with reasonable representation.

In this lecture we implemented LDA with different values for hyper parameters or the document topic prior and the topic word prior. Their effect on the document topic distribution and topic word distribution was observed respectively.

In this lecture, we use online learning approach for training the LDA model. The model learns incrementally as it produces topics based on the given data. Online approach is particularly useful when either you don't have access to all the data at once or the data is too big to be processed with given resources in a single step.

Topic modeling evaluation techniques focus on measuring the quality of the topics extracted by finding the compactness within the words chosen to represent a topic.

In this lecture, we calculate perplexity i.e., an intrinsic evaluation technique. The model is trained on a portion of the data and is tested on the remaining where the model with a smaller value of perplexity if favored to be a better model.

In this lecture, we introduce sentiment analysis which is applied to subjective data generated by a large number of amateur authors usually on social media and other online mediums.

In this lecture, we discuss the different techniques that can be used for sentiment classification. The quintuple of sentiment analysis is presented. Different applications and research problems consider a subset of these properties to focus on providing a solution.

In this lecture, we discuss the 3 levels at which a review document can be analyzed and what are the associated challenges with each one of these. The tools available have reasonable accuracy for building applications, while challenges are still actively addressed in research.

In this lecture, we make use of the subjective analysis with labels available to perform sentiment analysis as a classification problem.

In this lecture, we introduce wordnet as a dictionary that can be used for sentiment analysis

In this lecture, we perform sentiment analysis with the help of wordnet dictionary

In this lecture, we perform sentiment analysis with the help of SentiWordNet Dictionary

In this project, the user is allowed to enter an document, article, a query a review or anything. We pre-process the data and train our model on it. Next using KNN Classifier we identify the class label of the document entered by the user. We perform sentiment analysis using WordNet to suggest if the document entered by the user is positive or negative. Finally we also suggest the nearest documents using Nearest neighbors clustering to recommend to the user for further reading.

In this project, we did detailed pre-processing by removing everything that may not belong to the aspect (explicit aspect) or opinion (implicit aspect) category. Then the words are structured and using topic modeling 5 topics are identified from within the data. These topics are represented with their 10 most relevant words. Sentiwordnet is used to perform sentiment analysis on these topics.

These are the different application areas or ideas for the project. The students are expected to do a project where they would choose an application area, a type of data source and would build a model that would do something useful. Students are also encouraged to bring in their own ideas and discuss it with me for the project. Good Luck :)