Visualization and Imputation of Missing Data

Name: Visualization and Imputation of Missing Data
Rating: 3.5 (38 reviews)

Learn to create numerous unique visualizations to better understand patterns of missing data in your data sample.

Created byGeoffrey Hubona, Ph.D.

Last updated 9/2020

English

What you'll learn

Use visualizations created by R software to identify patterns of 'missingness' in data sets and to impute reasonable values to replace the missing data.
Recognize and identify the different patterns of missing data and the relative severity of their likely consequences.
Learn to use the VIM and VIMGUI R packages to create unique, novel and vibrant images which promote the understanding of patterns of both missing and imputed data in a set of data.
Learn the different historical approaches to impute reasonable values for missing data and their relative advantages and disadvantages.
Learn the characteristics of: (1) Hot-Deck; (2) K-Nearest Neighbor; (3) Regression-Based; and (4) Iterative, Model-Based, Stepwise Regression (IRMI) imputation techniques to "fill in" missing data and when and how to implement them with provided software.

Course content

4 sections • 38 lectures • 4h 53m total length

What is this Course all About?2:00
Explore patterns of missing data, learn to impute reasonable values using hot deck, knn, individual regression, and Urmi algorithm, and visualize missing and imputed data.
Introduction to Course and Materials7:40
The Allstat-Gui and Vim-Gui Applications3:29
What is Missing Data? (slides, part 1)8:35
Missing data are a common occurrence and can have a significant effect on the conclusions that can be drawn from the data. Missing datacan occur because of nonresponse: no information is provided for several items or no information is provided for a whole unit.
What is Missing Data? (slides, part 2)4:28
What is Missing Data? (scripts, part 1)6:57
What is Missing Data? (scripts, part 2)7:08
Approaches to Handle Missing Data (slides, part 1)8:15
In statistics, imputation is the process of replacing missing data with substituted values. When substituting for a data point, it is known as "unit imputation"; when substituting for a component of a data point, it is known as "item imputation". Because missing data can create problems for analyzing data, imputation is seen as a way to avoid pitfalls involved with listwise deletion of cases that have missing values. That is to say, when one or more values are missing for a case, most statistical packages default to discarding any case that has a missing value, which may introduce bias or affect the representativeness of the results. Imputation preserves all cases by replacing missing data with an estimated value based on other available information. Once all missing values have been imputed, the data set can then be analysed using standard techniques for complete data
Approaches to Handle Missing Data (slides, part 2)4:20
Approaches to Handle Missing Data (slides, part 3)7:33
Walk through Visualizations with VIMGUI (part 1)7:22
Explore visualizations and imputation of missing data using VIMGUI, including data, imputation, and visualization tabs, with four techniques including nearest neighbors, hot deck, and regression.
Walk through Visualizations with VIMGUI (part 2)5:49

Introduction to KNN8:52
In pattern recognition, the k-Nearest Neighbors algorithm (or k-NN for short) is a non-parametric method used for classification and regression.^[1] In both cases, the input consists of the k closest training examples in thefeature space. The output depends on whether k-NN is used for classification or regression:

In k-NN classification, the output is a class membership. An object is classified by a majority vote of its neighbors, with the object being assigned to the class most common among its k nearest neighbors (k is a positive integer, typically small). If k = 1, then the object is simply assigned to the class of that single nearest neighbor.

In k-NN regression, the output is the property value for the object. This value is the average of the values of its k nearest neighbors.

k-NN is a type of instance-based learning, or lazy learning, where the function is only approximated locally and all computation is deferred until classification. The k-NN algorithm is among the simplest of all machine learning algorithms.

Both for classification and regression, it can be useful to assign weight to the contributions of the neighbors, so that the nearer neighbors contribute more to the average than the more distant ones. For example, a common weighting scheme consists in giving each neighbor a weight of 1/d, where d is the distance to the neighbor.^[2]

The neighbors are taken from a set of objects for which the class (for k-NN classification) or the object property value (for k-NN regression) is known. This can be thought of as the training set for the algorithm, though no explicit training step is required.

A shortcoming of the k-NN algorithm is that it is sensitive to the local structure of the data.
VIMGUI Data and Aggregation Plot10:20
Customizing the Aggregation Plot4:23
Histograms and Barplots12:05
Spinograms and Splineplots7:53
Boxplots with Imputed Values (part 1)7:10
Boxplots with Imputed Values (part 2)6:54
Marginplots and Enhanced Scatterplots11:05

Introduction to VIM and VIMGUI (slides)9:36
VIM and VIMGUI Imputation Techniques (slides)11:32
Explore four imputation techniques for missing data, balancing speed and accuracy, including regression-based methods, mean and pairwise imputation, and k-nearest neighbors using vim gooey with gower distance for mixed data.
Introduction to K Nearest Neighbors (slides, part 1)10:14
Introduction to K Nearest Neighbors (slides, part 2)7:53
Explore k nearest neighbors for imputing missing data using distance-based estimates, including Euclidean distance and other metrics. Learn how the choice of k, weighting, and variables affect accuracy and smoothing.
Marginplot Matrix Visualization7:41
Scatterplot Matrix with Imputed Missings6:56
Parallel Coordinates Plot3:40
Matrix Plot6:01
Use a matrix plot to reveal multivariate dependencies and patterns, detect outliers, and visualize imputed values in a fully imputed sleep dataset.
Mosaic Plot8:02

Preparing VIMGUI Visualizations8:54
Begin Replicating VIM Aggregation Plot Example11:07
Continue Aggregation Plot Example8:21
Reproduce the imputation workflow in the gui by reloading the original sleep data, applying knm imputation with a chosen k, and examining how arguments and defaults shape the imputed dataset.
Finish Replicating VIM Aggregation Plot in VIMGUI10:32
Histogram and Barplot (part 1)7:19
Histogram and Barplot (part 2)5:28
Histogram and Barplot (part 3)12:29
Spinogram and Splineplot (part 1)6:51
reproduce spinogram and splineplot visualizations by imputing missing data with k nearest neighbors (k=5), then compare imputed patterns across air temp and humidity using multiple plots in gooey.
Spinogram and Splineplot (part 2)8:11

Requirements

Students will need to install R software but ample instructions for doing so are provided.

Description

There are many problems associated with analyzing data sets that contain missing data. However, there are various techniques to 'fill in,' or impute, missing data values with reasonable estimates based on the characteristics of the data itself and on the patterns of 'missingness.' Generally, techniques appropriate for imputing missing values in multivariate normal data and not as useful when applied to non-multivariate-normal data. This Visualization and Imputation of Missing Data course focuses on understanding patterns of 'missingness' in a data sample, especially non-multivariate-normal data sets, and teaches one to use various appropriate imputation techniques to "fill in" the missing data. Using the VIM and VIMGUI packages in R, the course also teaches how to create dozens of different and unique visualizations to better understand existing patterns of both the missing and imputed data in your samples.

The course teaches both the concepts and provides software to apply the latest non-multivariate-normal-friendly data imputation techniques, including: (1) Hot-Deck imputation: the sequential and random hot-deck algorithm; (2) the distance-based, k-nearest neighbor imputation approach; (3) individual, regression-based imputation; and (4) the iterative, model-based, stepwise regression imputation technique with both standard and robust methods (the IRMI algorithm). Furthermore, the course trains one to recognize the patterns of missingness using many vibrant and varied visualizations of the missing data patterns created by the professional VIMGUI software included in the course materials and made available to all course participants.

This course is useful to anyone who regularly analyzes large or small data sets that may contain missing data. This includes graduate students and faculty engaged in empirical research and working professionals who are engaged in quantitative research and/or data analysis. The visualizations that are taught are especially useful to understand the types of data missingness that may be present in your data and consequently, how best to deal with this missing data using imputation. The course includes the means to apply the appropriate imputation techniques, especially for non-multivariate-normal sets of data which tend to be most problematic to impute.

The course author provides free-of-charge with the course materials his own unique VIMGUI toolbar developed in the RGtk2 visualization programming language in R. However, please note that both the R-provided VIMGUI package (developed in RGtk2), as well as the course author's provided VIMGUI toolbar application (also developed in RGtk2) may have some problems starting up properly on a Mac computer. So if you only have a Mac available to you, you may have some initial difficulties getting the applications to run properly.

Who this course is for:

This course is useful for anyone analyzing large or small data sets that may contain missing data.
The course is useful for graduate students conducting quantitative, empirical research and/or practicing quantitative analytic professionals.
Please note that the VIMGUI software is written in the R-specific RGtk2 language (based on GTK+) which has been known to be problematic running on a Mac computer.

Visualization and Imputation of Missing Data

What you'll learn

Explore related topics

Course content

What is Missing Data? Imputation Approaches12 lectures • 1hr 14min

Missing and Imputed Visualizations8 lectures • 1hr 9min

VIM and VIMGUI Features; More Visualizations9 lectures • 1hr 12min

Reproducing VIM Package Visualizations in VIMGUI9 lectures • 1hr 19min

Requirements

Description

Who this course is for: