Python Data Science: Data Prep & EDA with Python

Explore what data science is, its workflow, and how it differs from machine learning, while examining roles, the data scientist skill set, and supervised and unsupervised learning with common algorithms.

Data science uses data to make decisions by answering what happened through descriptive analytics and business intelligence, and what will happen through predictive analytics and data mining, with data visualization.

Blend coding, math, and domain expertise to tackle machine learning problems. Cultivate soft skills like communication, problem solving, curiosity, grit, and googling to drive progress.

Explore how machine learning uses algorithms to learn from data and predict or find patterns, with supervised and unsupervised learning in data science.

Discover the most commonly used machine learning algorithms, including supervised methods like regression and classification and unsupervised methods such as clustering and dimensionality reduction, for data science practice.

Explore the data science workflow, including six steps—scoping a project, gathering data, cleaning and exploring data, modeling, and sharing insights—to deliver insights for end users and stakeholders.

Scope a data science project by identifying stakeholders, business problems, and required data, and determine where to find it; decide between supervised or unsupervised learning and assess necessity before proceeding.

Gather data from diverse sources, including flat files, spreadsheets, databases, websites, and APIs, to power analysis and ensure you collect the right data for your project.

Prioritize data cleaning to avoid garbage in, garbage out; correct data types, impute missing data, and reformat to ensure clean, accurate results.

Explore and understand cleaned data through exploratory data analysis (EDA) by slicing, dicing, and summarizing data across groups, then visualize with histograms and scatter plots to reveal insights.

Restructure data to input into a model by turning text fields into numeric features and performing feature engineering. Apply and tune machine learning algorithms for interpretable business insights.

Share insights with end users by revisiting the original problem, interpreting results, and recommending non-technical solutions that emphasize impact, or deploy the model behind a website for user-driven insights.

Explore data prep and exploratory data analysis within the data science workflow, covering project scoping, cleaning, and exploring data, and preparing data for modeling.

Learn how data science uses data to make smart decisions with supervised and unsupervised learning, and master data cleaning, preparation, eda, and the data science workflow.

Outline the process of scoping a data science project, from end-user needs to collaborating with stakeholders on tools, modeling techniques (supervised or unsupervised), and data requirements.

Define the project scope by clarifying goals, data sources, and techniques; think like the end user, and summarize the Maven Music project's scope and objective.

Think like an end user to guide data prep and eda by empathizing with stakeholders, identifying what matters, and selecting metrics to drive impactful analysis.

Think like an end user to brainstorm problems, not solutions, and identify why customers leave, guiding data prep and exploratory data analysis decisions.

Brainstorm solutions to customer cancellations by evaluating data-driven options alongside non-data approaches, highlighting data science benefits like data-backed insights and predictions, while noting time and resource trade-offs.

Learn to distinguish supervised from unsupervised learning in customer analytics across six scenarios, using historical data to predict outcomes, uncover themes, cluster customers, and visualize cancellation rates.

Identify data requirements by choosing modeling techniques, structuring data for supervised or unsupervised learning, and brainstorming inputs with end users, then locate data sources and scope the project.

Explore how data structure distinguishes supervised learning with labeled data from unsupervised learning without labels, and how X variables and Y labels guide modeling.

Identify features as model inputs to predict cancellations, and brainstorm inputs like listening history, monthly spend, demographics, and rate changes for supervised or unsupervised learning.

Identify data sources for model features, such as subscription history and demographics. Prioritize accessible internal sources, discuss access with end users and data warehousing team, and add external data later.

Define a focused data scope by starting with three months of internal subscription and listening history, prioritize quality over quantity, and iterate with end-user feedback toward a Python-based mvp.

Define the project scope with clear objectives and data needs, then use supervised learning on three months of subscription and listening history to predict cancellations and reduce them by 2%.

Think like an end user to define project goals, brainstorm boldly, and decide when data science is needed. Distinguish supervised (labeled data) from unsupervised models, and apply the MVP approach.

Install and explore your data science toolkit by setting up Anaconda and Jupyter Notebook, and learn Python basics, Notebook interface, and code versus markdown cells for data prep and EDA.

Discover why Python dominates data science with open source, scalability, and rich libraries for data prep, visualization, machine learning, and automation, backed by a large supportive community.

Install Anaconda by downloading the correct installer for Mac or PC, launching the pkg or executable, and accepting the defaults to complete the setup.

Launch Jupyter Notebook via Anaconda Navigator in your browser, then create a coursework folder and a Python 3 notebook in your documents, and keep the notebook server running.

Explore the notebook interface—the menu bar, toolbar, mode indicator, and code cells—and compare edit and command modes. Master saves, inserting and running cells, and keyboard shortcuts.

Compare edit mode and command mode in Jupyter notebooks: edit mode (green) writes code in cells, while command mode (blue) edits the notebook; escape switches modes, B adds cells.

Write and run Python code in a code cell of a Jupyter notebook. Compare code cells with markdown cells, observe execution counters and kernel status, and comment with pound signs.

Highlight markdown cells in Jupyter Notebook to document your workflow and format text using markdown syntax—headers with pound signs, bold/italic, bullets, numbers, code blocks, and links.

Discover Python resources, from Google searches and Stack Overflow to official docs and blogs, and learn to use Jupyter Notebook for data prep and EDA with code and Markdown cells.

Develop proficiency in Python data gathering by reading flat files and Excel spreadsheets, connecting to SQL databases, and saving data into a Pandas data frame for analysis.

Identify and locate data for analysis, read it into Python from various formats, clean and transform it, then store it as a clean Pandas data frame for analysis.

Explore data sources from local flat files and spreadsheets to databases and web access. Learn how Python reads and connects to SQL and NoSQL databases, and uses scraping and APIs.

Explore structured and semi-structured data formats such as tables, csv, and json, and learn how to read them into Python for analysis.

Explore the Pandas data frame, its zero-based index, and column data types, and learn to read data into a dataframe from external sources for Python data analysis.

Learn to read flat files with pandas using pd.read_csv, handling delimiters, headers, and file paths to load data into a dataframe in Python.

Learn to read CSV files in a Jupyter notebook using pandas read_csv, using relative paths, moving files into the Maven coursework folder, and adjusting separator and header arguments.

Learn how to read Excel files in Python using Pandas' read_excel, specify sheet_name to access zeroth or other tabs, and read the instructors and courses sheets.

Connect Python to a sql database using a driver and pandas read_sql to fetch data, with sqlite as the easiest option and mysql connector as an alternative.

Explore a Python data frame after import by using head, tail, shape, count, describe, and info to verify data and inspect numeric stats in a notebook.

Read a csv file into python, store the data in a data frame, and quickly explore the happiness survey data to confirm row count and happiness score range.

Learn to read a CSV into Python using pandas, create a dataframe, and quickly explore data with head, shape, describe, and info while checking for null values.

Learn how Python reads data from local flat files, spreadsheets, databases, and the web; organize data into tables with pandas data frames, then explore and prepare data for cleaning.

Learn how to clean data in pandas by converting data types, identifying and resolving data issues, and creating new calculated columns, transforming a messy raw dataset into a clean one.

Learn the purpose of data cleaning, converting raw data into analysis-ready formats by checking data types, addressing missing values and outliers, and creating new features, with a practical manual-cleaning approach.

Identify pandas data frame data types with dtypes and info, inspect sample rows with head, and learn to convert income to numeric and birth date to datetime for analysis.

Learn how to convert object columns to date time and numeric types using pandas to_datetime, handle missing values as NaT, and specify custom formats for nonstandard dates.

Convert currency-like object columns to numeric by removing dollar signs and commas with string.replace, then apply pd.to_numeric to yield floats.

Convert data frame columns from object to numeric by removing dollar signs, handling warm up time with coercion and string cleaning, and convert boolean rain values to integers using as_type.

Read Excel data into a pandas DataFrame, inspect column data types, and convert object columns to numeric or datetime for data preparation.

Guides converting data types in a cleaning workflow using pandas; read Excel data, inspect dtypes, convert alarm rating from object to numeric by removing stars.

Identify and resolve common data issues in your data frame before analysis, including missing values, inconsistent state abbreviations, duplicate rows, and outliers such as extreme incomes.

Identify missing data in Python data frames using df.isna(), then count per column with .sum() and inspect rows with missing values, noting nan, np.nan, and None representations.

Read the student grades data with pandas, identify missing values using nan, np.nan, pd.na, or None, and inspect non-null counts with df.info and df.isna().

Identify missing data and decide how to handle it. Apply options like keeping, removing rows or columns, or imputing values such as the median or city medians in Pandas.

Remove missing data with dropna on a data frame, exploring options like axis, thresh, subset, and inplace, with examples of dropping rows or columns and resetting the index.

Impute missing data in Python using the fill method, choosing median or mean appropriately, with examples for income and grades and in-place updates.

Learn to resolve missing data by updating specific values with df.loc based on domain knowledge, and handle multiple missing values using numpy.where for efficient data cleaning.

Identify missing data in the provided file and decide on handling approaches using best judgment, consulting the section five assignments notebook in the course materials.

Identify missing data in the sleep quality column using df.info and value_counts, then impute NaN values with two via fillna inplace and verify with updated info.

Identify inconsistencies and typos in text and numeric data using value counts for categorical columns and describe for numeric data, then clean class names and detect outliers.

Identify and fix inconsistent text using dot loc, update values with where, map categories, and apply string methods to normalize text (lowercase, remove punctuation).

Learn to update dataframe values with a logical condition using np.where and pd.where, including replacing Utah with UT and capping grades at 100.

Map values in Python with a dictionary to normalize data, linking keys to state abbreviations or labels, and demonstrate converting between data frames and dictionaries with map, merge, and np.where.

Clean messy text data by applying common string methods, including strip, lower, and replace, to standardize the location column, remove the word the, and produce cleaner data for analysis.

Practice data preparation by identifying and correcting text inconsistencies and typos while handling missing values in a Python data science notebook used in section five assignments.

Identify and fix inconsistencies and typos in the data by inspecting numeric ranges with df.describe, addressing 5.5 children as a non-typo, and normalizing activity levels to lightly active.

Identify and handle duplicate data in Python dataframes by using the pandas duplicated method, exploring how to detect, count, and view all duplicate rows with keep options.

Learn to remove duplicates with df.drop_duplicates, choose in-place updates or assign back to df, and reset the index to maintain a clean, merge-ready dataset.

Identify and resolve duplicate data in the section five assignments notebook to ensure accurate results in the Python data science tasks.

Identify and remove duplicate data by using df.duplicated to locate duplicates, then drop or keep non-duplicate rows to obtain 6,366 rows from 6,400 original.

Identify outliers via histograms, box plots, and standard deviation, and observe their impact on the mean. None definitively flags them; you decide as a domain expert.

Explore how histograms visualize data distribution and outliers in Python, using pandas df.hist and seaborn's histplot, and tune bins or bin width to reveal grade patterns.

Explore box plots to identify outliers and summarize descriptive statistics with median and quartiles using seaborn, calculating the interquartile range to flag outliers.

Calculate standard deviation to measure data spread, using mean and variance and numpy to identify outliers with three standard deviations away from the mean and example grades, via python code.

Learn four strategies to handle outliers in data prep, including keeping data, removing rows, imputing values (with min or max or average), and resolving via domain expertise in Python.

Review cleaned data by checking missing values, inconsistent text and typos, duplicate data, and outliers using df.isna, value_counts, and seaborn hist plot.

Identify and resolve outliers in the data frame, then re-explore the cleaned data after addressing missing data, inconsistent text and typos, and duplicates.

Identify and remove outliers using histograms and a box plot, then perform final data issues checks for missing data, value counts, duplicates, and overall data quality.

Create new columns from existing data to enhance analysis by applying numeric calculations, extracting date time components, and splitting text fields, as demonstrated with tax, month, and notes.

Create new numeric columns in Python by calculating percent spend from a numerator and denominator, derive the total spend, and use np.where to produce a with tax column.

Learn how to create new numeric columns in python data science using the groceries dataset, using square bracket notation, perform rounding, calculate percent inventory, and flag low inventory with NumPy.

Read pen sales data in a Jupyter notebook, check data types, and create numeric columns: total spend (pen cost plus shipping) and a free shipping indicator using conditional logic.

Create two new numeric columns from the dataset: total spend by summing pen cost and shipping cost, and free shipping with numpy where, after reading an Excel file with pandas.

Create new columns from datetime data by extracting components such as day, day of week, and time, and perform datetime calculations with pd.to_timedelta.

Create new columns from datetime columns by extracting time and day of week, then apply mapping to names and use time delta for date adjustments in a groceries dataset.

Apply python data science techniques to create a delivery time column by computing the days between purchase and delivery dates from DateTime data, then calculate its average.

Create a delivery time column by subtracting purchase date from delivery date using existing datetime columns, save as a new column, then compute the mean delivery time, about three days.

Create columns from text data by extracting characters using zero-based indexing, splitting text on delimiters, and flagging patterns with contains and regular expressions, then convert results into a data frame.

Demonstrate transforming text columns into numeric and categorical features. Extract prefixes, split values into category and subcategory, flag organic items, reorder columns, and save as a pickle file.

This assignment teaches splitting a text column into user name and review text using the pipe delimiter, and creating a boolean leak or spill flag for reviews.

Split the reviews column on the pipe to create username and review text columns, then assemble them into a dataframe. Flag leak or spill mentions with regex.

Check data types; convert object columns to numeric or datetime. Clean missing data, typos, duplicates, and outliers; create new columns; adopt a mvp mindset and start exploratory data analysis.

Explore data with exploratory data analysis (eda) in Python, learning techniques to filter, sort, group, and visualize data to uncover patterns and insights before modeling.

Explore data from multiple angles to uncover patterns and insights through visualization. Compare data cleaning with EDA and learn common charts like histograms and scatter plots.

Filter a data frame with a logical test using the loc accessor, apply and/or conditions, and contrast masking with loc versus square brackets, demonstrated with herbal teas and temperature.

Learn to filter a pandas data frame using lookups and square brackets, read pickle data, and build masks for multi-condition filtering, including low inventory and snacks.

Learn to sort a data frame using sort_values, with ascending by default, and reverse with ascending=False. Explore sorting by name and multi-column sorting.

Sort the groceries dataset by price with sort_values, show ascending and descending orders, sort by multiple columns (sub category then price), and chain a sort with a low-inventory filter.

Explore grouping data in Python using the split-apply-combine approach, group by type, and use .agg to compute mean, min, max for each group.

Explore grouping in pandas to summarize groceries by category and subcategory, using sum and count, reset index, and method chaining for clear, actionable data insights.

Filter the data to 2010–2019, group by country to compute each country's maximum happiness and identify the top five, then compute average happiness and compare the lists.

Explore data from the 2010s, filter for the 2010–2019 period, and compute each country's max and average happiness scores, highlighting Denmark as the happiest in the 20 tens.

Visualize data to reveal patterns, trends, and anomalies, using scatter plots to quickly uncover insights like sleep versus grades, and prepare to communicate findings to stakeholders with Python.

Explore quick built-in data visualizations in Python using Pandas dot plot on data frames to create bar and line charts without external libraries, demonstrated in a Jupyter notebook.

Demonstrates creating data visualizations in pandas, from dot plots and bar plots to line charts, by filtering data, sorting values, and pivoting data for multi-country trends.

Explore how seaborn pair plots visualize all pairwise relationships and distributions for numeric data with a single line of code. Learn to interpret the resulting visualization in your Jupyter notebook.

Demonstrate how to create a pair plot in Python with seaborn by reading data and selecting numeric columns, and interpret diagonals, scatter plots, and outliers.

Explore distributions in data columns after reviewing a pair plot, using frequency tables and histograms, and practice dot value counts and sns dot plots in Python.

Learn to create frequency tables and histograms for discrete and continuous data. Use techniques like value_counts, sort_index, rounding, and seaborn hist plots.

Explore the most common distributions in data science, including normal (bell-shaped) for continuous data, uniform, binomial, and Poisson, with emphasis on recognizing discrete patterns for machine learning.

Explore the normal distribution, defined by the mean and standard deviation, and apply the empirical rule—68% within one, 95% within two, and 99.7% within three—to guide interpretation and model preparation.

Import seaborn and build a pair plot to explore distributions of 100 song attributes, interpret the histograms for normality, skewness, and outliers, and validate interpretations.

Visualize spotify song data with seaborn pair plots to examine distributions, showing valence as normal, skewed energy, key and mode patterns, duration outliers, and tempo frequency analysis.

Explore scatter plots in seaborn to visualize relationships between two data columns, such as hours of sleep and test grades, and examine outliers after reviewing the pair plot and histograms.

Create a scatter plot with seaborn to explore the relationship between hours of sleep and test grades. Zoom in on outliers to identify patterns and consider actions based on data.

Use correlations to describe relationships between numeric columns and scatter plots; compute all-pairs with corr, identify weak to strong links, and remember correlation does not imply causation.

Explore how a correlation table and a pair plot reveal relationships between numeric columns, with hours of sleep versus hours studied illustrating strong correlation and near-zero correlation showing no relationship.

Explore correlations among musical attributes using the pair plot and scatter plots, then identify highly correlated or uncorrelated relationships and positive or negative trends in the Jupyter notebook.

Examine correlations using a correlation table and scatter plot; note no high correlations, that key is uncorrelated, and that energy correlates positively with valence while mode correlates negatively.

Export the Python data frame to CSV for use in Excel or Tableau, then create histograms with clear axis titles. Emphasize the audience takeaway, such as highlighting excessive coffee consumption.

Apply practical EDA tips by revisiting original questions, recording observations and outliers, and exploring with pair plots, histograms, and scatter plots, then subset and clean as needed before modeling.

Explore data from multiple angles with filtering, sorting, and grouping, visualize distributions and correlations in pair plots with histograms and scatter plots, and uncover insights for modeling and stakeholders.

Embark on a course project on movie ratings, applying gathering data, cleaning data, exploratory data analysis, and techniques to a Rotten Tomatoes dataset through filtering, sorting, grouping, visualizing, and aggregations.

Explore data prep by loading movie data, selecting columns, filtering to 2010 or newer, and analyzing Tomatometer and audience ratings for popular titles.

Create new columns for animation, action and adventure, and comedy by using genre text contains. Build a rating table by genre with groupby sums and compare animated versus non-animated ratings.

Visualize data with seaborn, create a pair plot excluding the last three columns, and note correlations between tomatometer and audience ratings, runtime with critic counts, and outliers like Shutter Island.

Learn how modeling-focused data prep differs from exploratory analysis, build a single input table, select appropriate row granularity, and apply feature engineering to prepare data for machine learning.

Follow a case study to prepare data for modeling and engineer features to predict dog food purchases. Create a table, define labels, and apply supervised learning for targeted email campaigns.

Transform data prepared for exploratory data analysis into a modeling-ready numeric table by converting to walk scores and splitting color into multiple columns; modeling means applying an algorithm.

Transform data into a model-ready format by aggregating into a single table, aligning row granularity, and ensuring non-null numeric columns, then engineer new features.

Learn to append and join tables in Python, vertically stacking identical columns or horizontally merging on a shared store column to create a single data table.

Learn to append data frames in Python with pd.concat to vertically stack tables in order with identical columns. Use axis=1 to horizontally combine frames and reset_index to 0.

Append data frames with pd.concat, handle matching and nonmatching columns, rename columns before concatenation, and reset the index to create a clean, vertically stacked dataset.

Learn to join data sets horizontally using merge, join, and concat in Python, focusing on left joins and matching on common columns such as store.

Demo shows merging sales and regions dataframes in pandas by joining on the store column, illustrating inner and left joins and using on, left_on, and right_on for multiple columns.

Learn the common join types in dot merge, including inner, left, right, and outer, explaining the how parameter, the default inner, and the Excel-like left join analogy.

Explore four common joins—inner, outer, left, and right—that combine happiness scores with population data in pandas, illustrating join on country columns and mismatch handling.

Create a single modeling table by left-joining the transactions and items tables on item_id, using python to merge data for the Maven Megamart case study.

Read four files in a Jupyter notebook, vertically append May and June sales to April, then join the book sales with customers on customer ID to form a single table.

Load four files into a Jupyter notebook, concatenate April, May, and June sales, and left-join with the customers table on customer_id to create a single final data frame.

Group data by customer to set row granularity, aggregate categories and spend, and build X and Y features to predict dog food purchases in June.

Prepare one-row-per-customer data for modeling by creating a june dog food purchases target and an april–may spend feature, converting types, and horizontally combining them into a clean modeling dataset.

Reformat the data to model customers, creating one row per customer, with a June purchases column and a total spend for April and May, then combine into a data frame.

Prepare rows for modeling by building a per-customer June purchases feature and an April and May total spend feature, then merge and fill missing values to create model data frame.

Ensure all columns are non-null and numeric before modeling, removing or imputing missing values; convert text fields with dummy variables and date-time fields with numeric calculations.

Turn text categories into numeric features by creating dummy variables, a form of one-hot encoding, to input color data into models. Use pandas get_dummies with drop_first to generate these features.

Convert the category text column into numeric dummy variables with pd.get_dummies, horizontally concatenate, aggregate by customer, and left-join back to the model for per-customer features.

Convert date time columns to numeric features by creating month dummy variables, days since last purchase, and average intervals between purchases using pandas get_dummies, groupby, diff, and shift.

Prepare date time columns by computing days since each customer's last purchase up to June 1st, then merge as a numeric days between feature into the model df.

Prepare columns for modeling by turning the audience field numeric, creating dummy variables, building three new columns for adult children and teen books purchased, and merging them with customer data.

Convert the audience column to numeric with get dummies, create three dummy columns for adult, children, and teen books, and merge into the model dataframe using customer id.

Explore feature engineering, creating new data columns to improve model predictions. See how features like total spend, pet supplies bought, and days since last purchase enhance modeling.

Apply a log transform to skewed data using numpy's log function to generate normally distributed features for modeling and feature engineering in data science workflows.

Apply feature scaling by normalization, standardization, or min max scalar to prepare data for models, normalizing to 0–1 or -1–1 or standardizing to zero mean and unit variance.

Explore proxy variables as a feature engineering technique to approximate missing or difficult to gather data, using median income or distance to the city center to improve house price modeling.

Apply domain expertise to craft features and use scaling and proxy variables to improve models; prioritize long data, RFM features for customers, and ongoing feature engineering.

Brainstorm features that would predict monthly book purchases. Then select two features, engineer them in Python, and add them to the non null numeric data frame for modeling.

Engineers two features: has school aged children and percent in person, using get dummies to convert text fields to numeric and build a model-ready data frame for June purchases.

Preview how data prep and EDA feed modeling by using a logistic regression model to predict dog food purchases from customer features.

Differentiate preparing for exploratory data analysis from modeling, join tables with dot merge and dot concat to a single non-null numeric table, create dummy variables and date-time transformations for modeling.

Read, clean, and explore Maven Music customer data in Python to identify factors driving churn, prepare for modeling, engineer features, and visualize insights for actionable recommendations.

Gather data for the final project by importing customer data and listening history with Pandas, exploring Excel tabs for songs and sessions to support churn prediction.

Clean and standardize data by correcting data types, imputing missing values, fixing typos, normalizing genres, and creating new columns such as canceled and clean email for effective EDA.

Perform exploratory data analysis to understand customer cancellations by membership duration and discounts. Join listening history with audio data to reveal listening sessions and genre popularity.

Prepare data for modeling and data analysis by building a customer data frame, computing listening sessions and percent of pop and podcasts, then explore correlations and pair plots for cancellations.