Data Cleaning in Python

In this lecture we discuss anomaly detection i.e., univariate outlier with the help of median. The median gives us a range of normality that we can apply to all values of a feature.

In this lecture, we implement the detection of missing values in the dataset.

In this lecture we use the median approach to define the range of normality for the detection of noise values.

In this lecture, we detect missing values with the help of median using the local context.

In this lecture, we discuss detecting noise values with the help of mean to determine the range of normality.

In this lecture, we implement the detection of noise values with the help of mean considering both the local and global context.

In this lecture, we discuss and evaluate the detection of a noise value or univariate outlier with the help of z-score.

In this lecture, we implement the detection of noise values using z-score based approach.

In this lecture, we discuss and evaluate the interquartile range for the detection of noise values.

In this lecture we implment the interquartile range to define the range of normality. Then used its lower and upper limits to compare all the values for the feature and identify noise values that are very much unlike the rest of the values.

In this lecture, we discuss the types of approaches that are used for handling or processing anomalies in the data particularly missing values and noise values i.e., univariate outliers.

In this lecture, we discuss the deletion strategy as an approach for processing records or features i.e., rows or columns that has got anomalies.

In this lecture, we handle the missing values with the deletion strategy i.e., to get rid of all the rows with the missing values.

In this lecture, we differentiate between the global and the local context for handling an anomaly. In case of global context, all the values of the dataset are considered to handle an anomaly while in case of local context i.e., concept restricted then only the instances that fall into the local context are used for handling the anomaly.

In this lecture, we discuss the replacement strategy for handling anomalies both with the global and local context.

In this lecture we discuss the use of statistical measures like median, mode and mean for handling missing and noise values.

In this lecture, we demonstrate how and when to use the mode value for the imputation of a missing or noise value in the dataset.

In this lecture, we make use of the statistical measures mean and median for the imputation of noise values in the dataset.

In this lecture, we discuss the multivariate outliers. These are the outliers that have more than one noise values in a record and the whole object or instance is considered to be an outlier. The types of multivariate outliers and their detection techniques are discussed. They are generally being filtered off instead of trying to fix their values for having issues with more than one feature.

In this lecture, we discuss the use of Local outlier factor for detecing multivariate outliers.

In this lecture, we implement and demonstrate the use of Local Outlier Factor for the detection of multivariate outliers in a dataset.

In this lecture, we briefly introduce clustering and a particular clustering technique that is very effective towards identifying multivariate outliers i.e., DBSCAN. Working mechanism of the algorithm is also explained.

In this lecture, we implement DBSCAN algorithm for clustering our data, however, our interest was in the points that are identified as outliers by the algorithm and are kept out of the clusters

In this lecture, we discuss how data visualization can be helpful towards detecting multivariate outliers in the data. We have also discussed how to come up with good features that can be used for the visualization purpose.

In this lecture, we implement the data by using the first two non-ID numeric columns of the dataset to plot a visualization of the data. It helped to manually inspect the outlier in the data.

This lecture introduces the various types of anomalies that can be found in textual data. It also demonstrates an example of what we want to achieve when textual data is passed through these normalization steps.

In this lecture, we implement the removal of whitespaces and punctuations while the text is converted to lowercase.

In this lecture, we implement the removal of stop words from the dataset using ENGLISH_STOP_WORDS list

In this lecture, we briefly discuss regular expression basics that should be sufficient for removing unwanted domain specific stopwords.

In this lecturew, we make use of regular expressions for identifying and removing the domain specific stopwords.

In this lecture, we implement the stemming and lemmatization of words so that the number of features can be reduced while words having similar meanings are grouped together to be represented as a single feature.

In this lecture we use the NLTK library for finding the parts-of-speech labels for the words in a sentence. POS has many other purposes as well. They are much more relevant in dialog and question-answeringn based systems. However, they are also frequently used for filtering the unwanted parts of speech. For example, in order to perform Sentiment analysis, the word types that are not expected to hold user opinions are filtered through its parts of speech rather than looking for each word individually.

In this lecture, we implement the separation of text into segments i.e., sentences and then segments into words or tokens. For this purpose, we make use of the NLTK library which is much more efficient in identifying tokens and segments as compared to splitting text on periods (.) and spaces.

The lecture introduces the need for structuring textual documents and converting raw text into numerical values that can be easily consumed my machine learning techniques.

In this lecture, we discuss the bag-of-words (BoW) approach for converting textual data into vectors. Its called BoWs approach because it loses the position related information about words and only retain their frequency related information.

In this lecture, we discuss the representation schemes of converting textual data into structured format by converting words or tokens to features and representing them through numbers for each document.

In this lecture, we take single document and use count vectorizer to convert it into a structured format. With such a small dataset, the students can see how the words become columns and the document hold a numeric value for each word i.e., feature showing its frequency of occurence in the document.

In this lecture, we extend our study to include more documents in the corpus and see how they can be converted into a structured format. The idea of starting with these small datasets is for students to cross check the numbers and verify.

In this lecture, we tune the parameters to find the suitable parameters and number of those parameters for training a good model.

In this lecture, we implement the TF-IDF based text representation scheme for structuring textual data. TF-IDF is the most commonly used text representation approach.

In this lecture, the process of vectorizing the textual documents is applied on a dummy dataset with few documents.

In this lecture, the processing of vectorizing the textual documents is applied on a dataset from the UCI Repository.

In this lecture, we discuss the need for feature scaling.

In this lecture, we discuss the normalization scheme for feature scaling and how it is affected by outliers.

In this lecture, we demonstrate the implementation of feature scaling using normalization strategy. It make use of the MinMaxScaler class from sklearn.preprocessing.

In this lecture, we discuss how to use standardization scheme for scaling the values of a feature.

In this lecture, we implement the conversion of feature values into their standardized values using StandardScaler from sklearn.preprocessing.

In this lecture, we discuss another feature scaling scheme that make use of the interquartile range and therefore, is not affected by outliers

This lecture, demonstrates the implementation of Robust Scaler using sklearn.preprocessing.

In this lecture, we use Pandas to deal with categorical ordinal features by replacing the string values with integers. The pandas function replace() for this purpose on the data frame having the dataset.

Categorical nominal features are converted into numeric features using one-hot encoding. This approach converts a feature into the number of features equal to the unique values of that feature. Among the newly added features for a column or feature, only one will have a value of 1 for a record and the rest will be 0.

Sequence encoding is used for integer representation of data as a sequence of numbers, so that the position and frequency based information of different data items could be retained. In order to use deep neural networks for text processing, the text is generally encoding as a sequence of integers. It requires  assigning unique integer values to specific data items and then replacing them accordingly. Further, truncation and padding is required to ensure that all documents / objects are of equal length.

• Deductive learning refers to the older expert based systems that required the human expert to provide all their expertise in the form of rules. The computer program would utilize those rules to make intelligent decisions. However, it was laborious and time-consuming. Inductive learning, on the other hand, require data and computations which were easily available, particularly due to the decline in hardware prices. In case of inductive learning, the model gets input and output data and learn the patterns from it that would be used for decision-making. The latter is also called Machine Learning.

In machine learning (supervised learning) to be specific, the model learns by finding patterns in the values of individual  features and their impact on the labels. If the value of a feature is on the higher side for one type of label and lower for another, it is preserved as a pattern. The prediction of multiple such patterns are combined to evaluate the label for a test case.

The lecture elaborates on the definition of machine learning to realize what is being considered when we claim the model is learning. How to model learn through experience or past data and how it is evaluated. The model attempts to improve its performance on the task using the evaluation criteria provided, iteratively until it cannot be improved any further in the given setup.

The lecture provides an overview of supervised and unsupervised learning. In case of supervised learning, the models attempts to find a pattern between the feature values and specific class labels. Whereas, in case of unsupervised learning or clustering in specific, the model evaluates similarity among the feature values of different objects.

Machine learning based solution may require a lot of tasks rather than just training the model on a training data. These tasks depend on the nature of data, its availability and the nature of the task. The general pipeline for building an end to end model that can be used to tackle a real-world problem is elaborated in this lecture. You may pick a specific technique for each module or step of the pipeline, or may skip some depending on their need.

In this lecture, we implement the acquisition of useful data from a webpage into python variables. At times it is desirable to prepare our own dataset as the publicly available dataset may not cover the kind of analysis that a user wants to perform. Therefore, data acquisition may also be a task before preprocessing it. Using the text based preprocessing that we discussed in the earlier module, textual data can be cleaned from code and other unwanted sequences.