Complete Python for data science and cloud computing

Discover NumPy and SciPy as the foundation of numerical Python for data analysis and machine learning, and learn how pandas enhances data frames for scientific computing, with Anaconda packaging.

Explore numpy arrays, including one-dimensional and multi-dimensional data, and perform operations such as slicing, indexing, reshaping, stacking, aggregations, and basic mathematical computations.

Create numpy arrays by combining mixed data types and understand how the final dtype is determined, with integers, floats, and strings guiding the result for data analysis.

Learn how to create numpy arrays with range objects and constructors. Understand data type rules and how start, end, and step define ascending and descending sequences.

Create numpy arrays efficiently by using zeros, replicate patterns with tuple multiplication, and convert objects with array-related functions, exploring element-wise multiplication and range-to-array conversions.

Demonstrate creating numpy arrays with a specified data type using the empty array constructor, covering integer, float, and string types, and print the results while inspecting the dtype property.

Explore multi-dimensional numpy arrays, build two- and higher-dimensional matrices from lists, inspect shapes, and perform element-wise and concatenation operations for data science workflows.

Learn how numpy array properties differ from functions, and use shape, size, dtype, and itemsize to inspect a two-dimensional array built from a range. Explore transpose via .T.

Slice one-dimensional numpy arrays by specifying a start index, an end index, and an optional step, with end treated as exclusive so the last included index is end-1.

Learn to slice numpy arrays by start and end indices with positive and negative steps, including reverse order, and extract 2d subarrays by selecting specific rows and columns.

Learn to slice numpy arrays using fancy indexing by passing an index array to extract a subarray in an order not the original consecutive order, for one-dimensional and two-dimensional arrays.

Explore fancy indexing in NumPy arrays by selecting rows and columns with index arrays, reshaping ranges into matrices, and reshuffling data to produce ordered or repeated results.

Explore NumPy arrays: transpose rows and columns, reshape data for multi-dimensional arrays, and compare in-place views versus out-of-place copies, including flattening methods.

Explore how numpy arrays can be reshaped and transposed, comparing ravel and flatten, and showing how views differ from copies and how reshape and resize affect memory and shape.

Learn to merge and stack numpy arrays, using concatenate and stack operations along rows or columns, with axis choices and shape considerations for data analysis.

Explore numpy arrays for analysis, including data processing functions, random sampling and loading text files. Learn to copy arrays, fill with a scalar, and reshape and slice data for analysis.

Explore numpy array data processing by converting to lists, updating by index, sorting by axis, computing first and second order differences, and locating nonzero indices and missing values.

Explore Numpy data processing functions, applying repeat to 2d arrays along rows or columns with axis, and using choose and take to index and assemble results.

Explore numpy data processing functions such as A.R.T., search sorted, where, compress, min, max, sum, cumsum, and mean to rank, select, and summarize data across 1D and 2D arrays.

Generate random datasets using uniform and normal distributions, integers, and samples from the random package. Set ranges, sizes, and seeds to reproduce data for analysis and learning.

Load and write data with numpy by reading external text files, specifying file name, delimiter, data type, and optional columns, skip headers, and then save results to a text file.

Learn how to load and save text data with NumPy, read flat files with delimiter handling, select columns, transpose data, and write back as tab- or comma-delimited files.

Begin the NumPy homework by tackling seven questions with sub-questions, and finish them one by one independently. Walk through each question and provide the answers in the next session.

Explore numpy basics by solving the first homework: create an array from 1 to 100, compute the mean and sum, apply sqrt to elements, and use diff for consecutive values.

solve the numpy arrays homework by converting a tuple to a numpy array, identify missing values using non-zero checks, count them, and return a sorted float64 array.

Learn to build numpy arrays from lists, convert ranges, and compose multi dimensional arrays. Explore shape, dimension, transpose, and copying with copy to produce identical arrays.

Generate numpy arrays from 0 to 4 and create repeated patterns using numpy repeat and a function to repeat the entire array, exploring one- and two-dimensional shapes, including reversing order.

Generate and manipulate NumPy arrays using zeros and ones, concatenate and shuffle them, compute percentages with mean, and extract non-zero indices and values from one- and two-dimensional arrays.

Introduce the second numpy homework, detailing eight big questions assembled from smaller items; complete all questions independently, with solutions to follow, focusing on questions 7 and 8.

Publish the solutions for the second NumPy arrays homework and walk through questions 7 and 8 using the video book.

Explore NumPy arrays and Python tools to generate data with uniform distribution, rank values with sorting, use where to select based on conditions, and compute the mean.

Classify data using a two-class discriminant analysis approach with numpy: read the three-column CSP file, compute per-class means, form centers, and assign new samples by Euclidean distance to centers.

Explore pandas series objects and their one-dimensional data structure, distinguishing labeled indexing from position-based access. Learn to create series from lists, arrays, or dictionaries for flexible data analysis.

Understand pandas data frames, including reading and writing CSV or text files, and performing aggregation, sorting, merging, and managing data types, indexing, and missing values.

Create Pandas data frames by constructing from arrays, lists, series, dictionaries, or reading data from sources. Define columns and indices, transpose, and inspect shape and values for data frame management.

Demonstrates creating pandas data frames from lists and dictionaries, setting columns, and exploring index and rows. Shows how to transpose using .T and inspect shape, size, and values.

learn to read external files into a data frame and control headers, missing values, selected columns, index, and data types, with path handling for linux and windows.

Learn to read external files into data frames with csv and text readers, handle missing values, set headers and indices, choose separators, and save results to csv.

Demonstrates reading external files into Pandas data frames using Jupyter notebooks. Use absolute and current directory paths, read csv and flat files, and apply chunked reading for large data.

Learn to convert data types in a pandas dataframe, handle encoding, clean text by removing symbols, and impute missing values such as replacing prices with zero.

Demonstrates data conversion in data frames using map for a column and apply for the frame, converting price to string and back to numeric, with missing values imputed to zero.

Learn how to perform arithmetic operations on data frames using scalars, series, and other data frames, with column name matching and handling missing values.

Apply arithmetic operations on data frames using scalar values, series objects, and cross-frame methods by matching column names to add, multiply, or divide spend and income while handling missing values.

Learn to subset data frames using position and label indexing, apply logical filters (income > 60000, age 40–60), and select specific rows, columns, or scalar values.

Select a single column as a series from a data frame by name with single brackets, and use iloc or loc to slice by index or label.

Explore how to slice a data frame by rows and columns using position-based and label-based indexing, including selecting subsets with conditions like income and gender, illustrated on a coffee dataset.

Show how to slice data frames and series using index values and labels, compare single and double bracket outcomes, and apply boolean and type-based selections to extract numerics and strings.

Learn to manipulate data frames by creating new features such as ratio under price and weight conditions, joining and concatenating frames, updating and renaming columns, and dropping columns.

Learn to manipulate data frames by adding, deleting, and updating rows, using axis to operate on rows or columns, and applying append and concatenate with inner or outer join.

Learn to remove rows from a data frame using boolean selection and slicing, with drop and conditional checks, then update a single cell via label-based and position-based indexing.

Learn to update and manipulate data frames in Python using conditional logic, new features, and techniques like slicing, moving averages, and concatenation to build and merge datasets.

Rename and drop columns or rows in a data frame, with in-place options. Utilize map and apply to update series and data frames, add new features, and insert columns.

Sort and rank data frames with sort_values and sort_index, choosing ascending or descending order and in-place or new frames. Sort by column values or labels, use multiple keys.

Rank data frames using axis-based ranking, choose ascending or descending order, and handle ties with methods like average, minimum, maximum, or first. Explore grouping to compare patterns across income-based groups.

Sort data frames by values and by index label in Python, including sorting by customer ID, gender and age, and choosing in-place versus new data frames.

Learn how to assign ranking memberships in a data frame with rank, distinguish ranking from sorting, and create ranking groups (quintiles) using cut and qcut.

Explore q cut and the cut function to bin a variable into four categories, either evenly distributed or density-weighted, with customizable midpoints and labels.

Learn to combine data frames in pandas by row and column stacking, and merge them with inner, left, right, and full outer joins using common keys.

Demonstrate stacking, concatenation, and merging data frames in Python, using inner and outer joins, common keys, and handling missing values across sample datasets.

Learn how to set and manage indices in data frames, including regular and automatic indices, reset and re-index operations, and hierarchical indexing for efficient querying.

Explore how to set and use an index in data frames to speed searches, distinguish between index columns and regular columns, and reset indices after aggregation.

Explore indexing methods in data frames with re-index versus index, add a new first name column by forcing extraction from the original index, and handle missing values and common errors.

Demonstrate hierarchical searching with a two-column index on education and gender, using levels and top level to search for college or female while controlling which fields are returned.

Explore pandas indexing methods for data frames, including reset_index, in-place operations, reindexing, and hierarchical index levels, with practical examples using gender and education data.

Learn to reshape data frames by transforming vertical data into horizontal shapes using transpose, pivot, stack, unstack, and melt, with practical city temperature examples.

Learn to reshape data frames by transposing long to wide formats using pivot, stack, and unstack, turning city-season data into columns of temperatures.

Explore how to reshape data frames from long to wide and back using pivot and melt, generating a transaction count by customer and store department in a retail transactions dataset.

Learn to clean data frames by handling missing values and duplicates, choosing methods based on missing data patterns, and applying imputation, indicators, and external data mapping.

Identify and treat missing values in data frames with two core functions, replace or impute missing data, and decide when to drop rows or columns.

Impute missing values in data frames using mean or median, apply fill methods (forward fill, backward fill) with optional limits, and create missing-value indicators to capture missingness.

Identify duplicates in data frames and use a boolean series to flag them. Drop duplicates either across all columns or by specific columns, noting data loss when rows are removed.

Learn to recognize and treat missing and duplicated values in data with Python, assess their impact on regression analyses, and quantify missingness using counts and sums to guide cleaning strategies.

Learn to treat missing and duplicate values using drop functions, how and axis options, and thresholds to drop rows or columns based on missing value patterns and percentages.

Learn to handle missing values by creating dummy indicators for missing data to preserve information, and apply mean-based imputation with an imputer to generate a dataset with imputed values.

Identify and quantify per-id duplicates in a data frame using shape and unique, then drop duplicates while keeping the first to clean the dataset.

Identify duplicates with value_counts and boolean indexing by id; compare first and last records using groupby to detect differences in other columns, and decide when to drop duplicates cautiously.

Learn how merging datasets generates non real missing values, interpret their meaning, and compute target rate with mean and other statistics, then clean data by removing missing values.

Learn to summarize and analyze data with pandas data frames using descriptive statistics: mean, median, count, value_counts, describe, covariance, standard deviation, and variance to reveal patterns and insights.

Explore using pandas group by objects to compute descriptive statistics—mean, min, max, median—across groups such as gender or age, and apply aggregation to multiple variables.

Use an anaconda notebook to summarize a 20,000-record, 17-column demographics dataset and derive statistics for population and groups, enabling market segmentation and city-id validation through counts and unique checks.

Generate random samples from a population using a sample function with a uniform distribution. Explore random seeds, non-replacement sampling, and computing means with group by and rounding.

Analyze mortgage to rent ratios with Python by sorting data to find top five states. Group by city to compute mean values and identify highest averages using idx max.

Use the unique function to list states, then group by state and city and count records to show city totals by state; inspect numerical fields for missing values.

Compute the median debt by state for cities with more than 10, using filter and group by to shape data, then calculate the median and sort to identify top states.

Explore how to create age groups with the cut function, generate a ranking membership variable, and summarize data by state and age group using means of female and male counts.

Group data by state and age group to count observations and missing values, then create education rank and generate dummy variables with get_dummies and one-hot encoding.

Learn to bin continuous data into age and income groups using cut and qcut, compute group-wise mean or median, and explore age–income relationships with apply on data frames.

Convert categorical variables to dummy variables using get_dummies to create 0/1 indicators with a prefix, enabling analysis of attributes like employment and education in a data frame.

Learn to convert categorical columns to binary using dict vectorization, generating sparse or dense arrays and reconstructing data frames for scalable data analysis with many categories.

Apply label encoding in scikit-learn to convert categorical strings into integers, then use one-hot encoding to produce dummy variables for large datasets.

Explore categorical data analysis using label encoding and one-hot encoding to convert education categories into dummy variables, create new features, and handle sparse matrices in Python.

Learn how sparse matrices efficiently represent large data with mostly zeros, convert categorical variables to dummy variables, and merge with original data for machine learning tasks.

Learn to load and save data frames from diverse data sources using pandas, including Excel files, CSV, and SAS, with read and write operations and grouping by mean.

Install and configure Python libraries to access SQLite, read data from a file into a data frame, connect to database, and query and summarize consumer data by education and gender.

Scrape web site data with Python by sending requests to dynamic pages, extract option information from prudential.com and Yahoo Finance, and format results into a pandas data frame.

Learn to scrape a web table with pandas and requests, convert it to a data frame, label columns like city, state, institution, closing date, showing 557 rows and 6 columns.

Tackle the first panda's data frame homework with about 50 small questions across three big questions, and download the solution in a Jupiter notebook program for comparison.

Explore the first pandas homework solution with python code packaged in a notebook file for use in Anaconda Jupyter; finish your work independently and compare with the provided solution.

Complete the second pandas homework by downloading the pdf and finishing all questions independently, then watch the video walkthrough to compare your solutions with the provided answers.

Perform year-over-year sales analysis with pandas by transforming dates, filtering for a product, and plotting a bar chart; identify the top five salespersons by total sales.

Learn to use Python with MongoDB for fast data ingestion and analysis, leveraging a document-oriented database with JSON-like structures.

Download and install MongoDB on Windows, set up data and log folders, start the service, verify via command line, and use Python to access MongoDB for data analysis.

Connect Python with MongoDB using PyMongo, set up a localhost:27017 connection, and perform insert, find, and filter operations on the people collection, highlighting key differences from relational databases.

Insert many records into mongo table from a pandas data frame using insert_many, then read back into pandas and apply aggregate to compute averages by gender and sums by education.

Explore data visualization with Matplotlib as a foundational tool, generating bar, pie, and line charts from pandas data, using grouping and mean calculations with titles and legends for clear insights.

Explore matplotlib line styles, build line and bar charts (stacked and side-by-side), histograms with KDE curves, scatter plots with a regression line, and the scatter score matrix package.

Introduce seaborn, a plotting package built on top of Matplotlib that yields smoother, more beautiful charts and supports custom color palettes; install seaborn with pip and upgrade when needed.

Demonstrate seaborn data visualization in Python with strip plot, swarm plot, box plot, bar plot, counterplot, and point plots to compare employment and income in coffee data.

Demonstrates data visualization with seaborn to explore a car price dataset using univariate analysis, correlation, and linear regression lines. Visualizes distribution plots, count plots, and swarm plots to reveal relationships.

Install ggplot for python with pip, and explore its one sentence one plot design and layer-by-layer grammar to plot data with variables and add layers via the plus operator.

Install Plotly and learn to generate interactive plots, with offline and online options to host, embed, or download graphs on the Plotly online platform.

Explore offline Plotly to create scatterplots, line graphs, and bar charts using graph objects, traces, and layouts, and generate interactive plots locally.

Demonstrate creating an offline Plotly scatterplot from random data, customize markers with color and size, and configure the layout and zero-based y-axis.

Learn to generate interactive visualizations with Plotly both online and offline, store graphs on your website for future access, and authenticate using an API key after registering.

Learn to generate and use an API key for online Plotly to render a scatter visualization, manage credentials and datasets, and display the plot on the Pratley website.

Explore and interpret statistical tests in Python data science, covering distributions, density charts and frequency charts, correlation and linear regression, t-tests, chi-square, nonparametric tests, p-values, confidence intervals, and bootstrapping.

Test whether the population mean equals a fixed value using a one-sample t-test or a normal distribution test, based on normality, and check normality with a histogram and q-q plot.

Explore p-values and t-tests to compare means using one-sample and two-sample designs. Determine when to reject the null hypothesis, based on sample size and variance.

Apply a two-sample t-test to compare spending between test and control groups after a campaign, and interpret the p-value to assess significance.

Apply nonparametric tests in Python when data are not normally distributed, focusing on the rank sum test. Rank observations, compute the U statistic, and interpret p-values to compare two groups.

Introduce analysis of variance (ANOVA) as the extension of the t-test for multiple groups, outlining normality assumptions and how ANOVA compares group means while reducing confounding factors.

Use anova to compare education groups on income. Calculate between-group sum of squares, grand mean, and F statistics to obtain a p-value for testing H0.

Explore how analysis of variance uses education level to explain income variance, decomposing it into between and within components via mean square and the f-statistic for p-values.

Conduct a one-way anova to compare collection rates across four groups and determine whether differences exist, interpreting the f statistic and p-value to identify which group differs.

Explore one-way and two-way ANOVA concepts in Python, assessing education and gender effects on income, with linear regression, interaction terms, dummy variables, and p-values for significance.

Explore interaction effects in data analysis by comparing how gender and age influence spend and income, and distinguish between correlation and interaction using anova and modeling concepts.

Explore how ANOVA and ordinary least squares in Python handle two categorical factors: home ownership and education levels, and interpret interaction effects and p-values to explain spend on food.

Examine repeated measures by comparing before and after results on the same subjects, using paired tests (paired t-test or sign-rank tests), calculating mean differences, their standard deviation, and p-values.

Explore paired tests for data science: use the sign rank sum test for nonnormal paired data and the paired t-test for before–after or multi‑timepoint comparisons, including differences, ranks, and p-values.

Explore how to test relationships between categorical variables using the chi-square test, interpret p-values, and compute expected frequencies from contingency tables.

Perform a chi-square test on two categorical variables, interpret the chi-square statistic and p-value, and examine the contingency table, expected values, and odds ratios with their confidence intervals.

Explore proportion tests by comparing two sample proportions to assess equality using a z-test, normal distribution, and central limit theorem, with examples such as coffee consumption in two groups.

Explore how to perform and visualize statistical tests in python for data science and cloud computing, using anaconda notebooks, including chi-square and one-sample proportion tests, with seaborn plots and p-values.

Apply python-based statistical tests to credit risk data, examining relationships between delinquency and variables such as age and 90-day delinquency counts, with proportion and non-parametric tests.

Chi square reveals a dependency between 90 days delinquency and serious delinquency in two years, and age serves as a predictor via nonparametric tests on zero vs nonzero delinquency groups.

Learn how to use Python to perform statistical tests and compare groups. Check normality with Q-Q plots and density curves, and interpret linear regression results.

Using the cloud, this lecture demonstrates a normal distribution with a density function, filters income data, and tests whether the mean is 44000 with a t-test, despite imperfect normality.

This lecture demonstrates performing a one-way statistical test (ANOVA) in Python to compare age across three employment groups—full time, part time, and student—and interpret a significant difference.

The lecture demonstrates how to perform a paired t-test on related features, comparing income vs spend and cups of coffee vs tea, using one-sample and nonparametric tests on differences.

Explore how to conduct ANOVA with multiple categorical variables in python, using data cleaning, ordinary least squares, and interpretation of p-values alongside interaction effects on income and spend on food.

Explains using chi square tests and p-values to test differences and dependencies, then conducts a correlation study on merged data, creates a feature, and evaluates Pearson correlations for predictor selection.

Calculate the correlation coefficient and explain it with pocket analysis by grouping income into ranking packets (income_drp) and using a bar chart to show the negative relationship with the target.

Download the pedia file and complete all statistics questions independently, using the statistical graph. Review Anaconda Jupiter notebook solutions, download the notebook, and compare your answers after solving each question.

Explore the linear regression model with training data to predict outcomes from predictors, including simple and multiple regression, coefficients, least squares, and scatterplots.

Explore simple linear regression with a single predictor x and response y, explaining the intercept and slope, and validating via residuals' normality and independence checks.

Learn how multiple linear regression extends simple regression with many predictors, interpret R-squared and adjusted R-squared, and guard against overfitting using error decompositions and residual analysis.

Explore linear regression with least squares to estimate a, b, and c, compute SST, SSR, and SSD, and evaluate with R square while predicting outcomes from sample data.

Explore feature selection and feature engineering to enhance linear regression, using low-variance filtering, univariate selection, regularization, new feature creation, dummy variables, and interaction terms to improve model performance.

Explore three feature selection methods, variance threshold, univariate chi-square selection, and Lasso, demonstrated on datasets, reducing features from three to two, four to two, and four to three.

Explore feature engineering by creating new features from existing data to boost model performance, including utilization feature from balance and credit limit and the use of dummy variables via binning.

Explore logistic regression by predicting owner versus renter using income and age as predictors from a training dataset, and apply the model to classify new customers.

Apply logistic regression to binary targets to predict the probability of owner versus renter using income and age, contrasting with linear regression for continuous targets.

Explore the logistic regression model for classification using income and age to predict owner probability, apply probability thresholds to classify owners versus renters, and understand the logistic curve.

Apply logistic regression to map a linear combination of predictors with an intercept to a probability via the logistic function, yielding probabilities between 0 and 1 for binary outcomes.

Estimate logistic regression coefficients from training data using a binary distribution and the likelihood function. Apply supervised learning to predict future data, noting that unsupervised learning lies beyond this section.

Maximize the log-likelihood to estimate logistic regression coefficients, then predict the probability of owner as a binary outcome using income and age.

Explain logistic regression by modeling the log odds as a linear function of predictors and interpreting coefficients as changes in odds ratios, illustrated with age and income.

Explore logistic regression for binary classification, converting predicted probabilities into owner or renter labels using a threshold, and evaluate with a confusion matrix, sensitivity, specificity, and accuracy.

Explore logistic regression metrics, including accuracy (72%), sensitivity, specificity, AUC, KS, and false positive rate and false negative rate at a chosen threshold.

Explore how to compute sensitivity, specificity, and ROC AUC from logistic regression predictions, interpret cross-tab results, and assess model performance for owner versus renter classification in credit risk contexts.

Explore logistic regression validation through decile analysis and lift, translating predicted probabilities (0–1) into groups to compare group ownership percentages with the overall average.

Learn how to validate a logistic regression model for binary ownership, using probability scores, thresholds, and 10-cell analysis to interpret performance metrics like accuracy and false positive rate.

Explains logistic regression for targeting, demonstrates lift across deciles, and clarifies predictive value. Uses Lorenz curves and sensitivity and specificity curves to assess separation power against random baseline.

Analyze a practical linear regression model to predict weight from workout time and food intake, with normality checks and interpretation of R-squared and p-values.

Explore the use cases of statistical models by assessing a linear regression with p-values, coefficients, and R-squared, using statsmodels to identify significant variables and validate residuals.

Explore linear regression with many variables, focusing on feature selection and correlation to predict customer transaction values. See exploration, missing value checks, and univariate correlation analysis to identify important variables.

Rank correlations to remove the bottom 25 percent and keep the top 300 variables; apply lasso with range-standardized data to select significant predictors, keeping target and id.

Apply lasso-based feature selection to identify important variables using coefficient thresholds and percentiles, fit a final linear model, and generate predictions for the top records.

Apply lasso-based feature selection to fit a linear regression model on the training data and evaluate it with r-squared and log-transformed error, noting 11 percent r-squared.

The lecture explores correlation study using Pearson correlation, feature selection to pick top variables, and compares linear regression with the lasso method, highlighting R-squared outcomes and Python code-based calculations.

Explore how logistic regression uses age and income to predict ownership probability for a binary target, and learn data preparation, model fitting, and probability prediction.

Validate a logistic regression model using predicted scores and ranking groups, generate ranks with qcut, and assess performance by comparing average target and score.

Generate a bar chart of average target and average score across ranking groups using the Tessera table, then compute lift against the overall mean.

Learn to evaluate logistic models with ROC curves and AUC, compute false positive rate and true positive rate using scikit-learn, and understand overfitting risks and KS statistics.

Complete the homework on a linear regression model using the car price.csp data, download the CSP file, follow the question list, and review the lecture for solutions.

Walk through solving a statistics homework in python using pandas, covering data import with na values, missing data checks, filtering positive values, creating dummy variables, concatenation, and median imputation.

Learn how to prepare price data for linear regression by checking normality with plots, truncating long tails, standardizing features, selecting top correlated variables, and evaluating with p-values.

Select the independent variables and the price from the data frame as dependent variable, fit linear regression, and validate the model with p-values, r-squared, mean squared error, and diagnostic plots.

Explore fraud detection using credit card transaction data to predict fraud probability with logistic regression, decision trees, and random forest, emphasizing handling numerous categorical variables.

Manipulate data, slice datasets, and perform feature engineering on transaction records to predict fraud using a binary target and chi-square analysis on an Excel dataset.

Load the dataset into a dataframe, inspect shape and features, and rename columns to id and target in Python. Create a target from yes/no and check missing values with sum.

Explore handling utf-8 encoding, inspecting dtypes, extracting character columns, and building a cross-tab with chi-square tests to assess predictors for fraud in a data science workflow.

Explore univariate analysis for fraud detection by evaluating categorical variables with chi-square statistics and p-values, then assess numeric predictors via logistic regression and AUC performance for feature selection.

Perform feature engineering by creating dummy variables for each categorical variable, rename columns to avoid duplicates, and use get_dummies to expand the dataset with 48 new variables.

Learn to clean fraud data by dropping duplicates on transaction IDs to guarantee unique IDs, apply range standardization (min–max scaling) to numerical features, and combine dummy variables for modeling.

apply chi-square univariate analysis to select 18 variables for the fraud detection project, then use a selector to transform data and identify the chosen feature names.

Split data into training and testing sets, apply cross-validation, and train a fraud-detection model evaluated by AUC and accuracy, about 85% AUC and 82% accuracy.

Learn to use historical data to predict monthly online product sales, including campaign impact, transforming categorical variables prefixed with c.a.p into dummy variables and applying a random forest with cross-validation.

Predict online product sales using historical product and campaign features to forecast monthly outcomes (m1, m2). The lecture covers dataset structure, missing-values checks, and imputation for independent variables.

Explains Python codes for predicting online product sales by handling missing values with missing-dummy indicators, median imputation, and reshaping data from wide to long using stack and transpose.

Explore Python data prep for predicting online product sales: merge frames, handle missing values, encode categorical variables with label and one-hot encoding, and use sparse matrices for modeling.

Explore feature selection using tree-based importance to reduce 2000 features to key predictors. Build and compare models including random forest regression and gradient boosting regression to predict online product sales.

predict credit risk and delinquency probability from historical credit data to derive a credit score, using factors like utilization, income, mortgage, and demographics.

Explore how to build a credit risk model using logistic regression in Python, including data preprocessing, cross-validation, scorecard construction, and missing-value imputation.

Analyze missing data in credit risk dataset with data slicing, identify variables with missing values—monthly income and number of dependents—compute the missing percentage, and outline imputation or dummy variable options.

Identify the missing percentage from 150000 records, create binary missing indicators for income and dependent information, and impute missing values using the mean or other constants.

The lecture covers building a credit risk scorecard by cleaning data, binning variables to capture nonlinear patterns, and identifying key predictors like utilization, delinquency bands, age groups, and credit lines.

The lecture demonstrates a Python-based correlation study using Pearson correlation to rank variables by their impact on delinquency, selects top features with a 2% threshold, and builds a data frame.

Selects features from X and trains a logistic regression model with Lasso regularization, using 5-fold cross-validation to average AUC and ROC/PR performance.

Develop and validate a logistic regression credit risk scorecard in Python, using cross-validation and AUC, then split data into 70/30 training and validation sets to prove predictive value.

Learn how to partition the validation data into ten ranking groups, generate a scorecard model, and evaluate it with lift, cumulative lift, KS, and related statistics.

Develops a Python-based credit risk analysis workflow that builds features, computes group-wise cumulative good and bad counts, and evaluates model performance with lift, KS, sensitivity, specificity, and Lorenz curves.

Build a credit risk model with lasso, select non-zero coefficients, use ROC and confusion matrices, and threshold at 0.5 to reach about 93 percent accuracy.

Analyze the impact of sales promotion offers using a time-series regression with control and testing groups, dummy variables, and interaction effects to drive data-driven conclusions.

Compare sales between control and testing groups to assess campaign impact; impute missing values, transpose data to long format, and align transaction records by time points.

Analyze manipulated data to compare sales between the control and testing groups under a campaign. Define seasonality factors and convert months and year into categorical variables for analysis.

Convert months and year into dummy variables, concatenate them, and apply log transformation to normalize sales data; then build and merge control and testing groups for promotion impact analysis.

We compare testing and control groups with a t-test to detect significant sales differences, and trim data at 1.5 to enable regression modeling based on a normal distribution.

Apply linear regression to measure the sales promotion offer's impact, interpret coefficients and p-values for both numeric and categorical variables using statsmodels, revealing a medium positive effect on sales.

Analyze sales promotion effects by comparing testing and control groups across years, identify interaction between factors, generate an interaction term, fit a model, and interpret p-values and r-squared for significance.

Analyze how product descriptions from transaction records predict price and reveal price elasticity and demand relationships through text mining, stopwords removal, and feature extraction.

Learn how bag of words converts text into vectors using term frequency and inverse document frequency, enabling feature extraction for machine learning on document collections.

Explore how price and demand relate through price elasticity, using linear regression to model sales with price, promotion, product type, seasonality, and other factors.

Explore the PYENSON code to predict retail prices from transactional data, using product features, purchase behavior, and NLP to encode structured, categorical variables, and analyze price elasticity.

Clean price data by removing commas and converting to float with a log-scale transformation. Aggregate transactional data by categorical variables, then compute mean price and quantity to predict price.

Leverage scikit-learn text processing to predict prices from item descriptions, using tf-idf and count vectorizer features, with stop-word removal, stemming, and regex-based cleaning to reduce noise.

Explore how tf-idf vectorization converts text into a sparse feature matrix, selects top features by frequency, and exposes feature names for NLP-based price prediction.

Explore extracting tf-idf features from a document collection to identify important words, build a sparse matrix, and convert results into a data frame for model-ready features.

Learn how to apply tf-idf features to rank words by importance for NLP-driven price prediction, filtering low-idf terms and extracting the most influential terms from a document.

Remove rare dataset categories and validate data to reduce noise in price prediction with NLP. Separate item descriptions as text features and group numerical and categorical variables for modeling.

Master essential natural language processing text preprocessing, including stemming to unify words such as produce and product, and stop words removal followed by tokenization with regex patterns.

Describe an NLP preprocessing pipeline with tokenization, stemming using the English stemmer, stop-word removal, punctuation removal, and lowercasing words across documents.

Learn to prepare text data for price prediction by building tf-idf features from a corpus, vectorizing documents, and using fit-transform to create a sparse matrix for NLP-driven models.

Build a python data pipeline that generates 60 tf-idf features, creates dummy variables for categorical variables, merges sparse and numerical data, and selects the 300 features most correlated with price.

Split the data into training and testing sets, train a linear regression model with tf-idf and normal features, predict prices, and evaluate with MSE and MAE.

Develop a price elasticity model using a dataset to estimate how a 1 percent price increase reduces quantity, incorporating seasonality and price coefficients.

Clean and transform the data by removing records with missing year or month, keeping years 2010 and 2008, converting features to float, adding a natural ID, and resetting the index.

Learn to build a pricing model and elasticity estimate in Python by selecting items with multiple prices, merging data, applying log transforms, and fitting a linear regression with dummy variables.

Select top correlated variables from a prior study and fit a linear regression to predict quantity from price, checking p-values and r-squared; estimate category price elasticity.

Build a customer and product recommender system using customer segmentation on training and transactional data. Apply the trained model to the population to recommend the right products to each customer.

Develop a customer and product recommender using a training data set, selecting key products by transaction frequency, classifying customers as purchasers or non-purchasers, and applying kamins segmentation to tailor recommendations.

Learn how to build a customer and product recommender in Python by loading a combined train-test dataset, selecting features, forming segments, and identifying top products by frequency for each segment.

Use python and pandas to build a customer and product recommender by merging a key product dataframe, filling missing values with zero, and classifying buyers versus non-buyers through per-customer aggregation.

Define buyers and non-buyers, merge with the total transactional data via outer join, fill missing with zero, compute per-product transactions and frequencies, and rank by buyer-to-non-buyer ratio.

Explain python codes: customer and product recommender with functions to select initial key products for each segment and a rolling procedure to add more products under a transaction-based threshold.

Explains building initial key product lists per segment, iterating with a frequency threshold of 200, imputing zeros, and clustering customers to assign segment membership.

Open data set, extract training data to train cluster memberships, and use a rolling process to add key products per segment until a threshold, with Kamins procedure and cross-tab validation.

Explore how Spark and Hadoop enable big data analysis across multiple nodes in the cloud, using map reduce, cluster managers, and executors to split data and speed up processing.

Learn how Spark and Hadoop integrate for data analysis, using Spark core and RTD in-memory processing with transformations and actions, and leverage PySpark for Python-based data science tasks.

Discover Amazon Web Services (AWS) for data science and cloud computing, covering EC2, S3, and EMR, plus Spark, Presto, Zeppelin, and Glue for scalable data preparation and analysis.

Learn how to register for a free AWS account, complete the signup form, enter payment details, review the usage limits, and log in to get started with AWS services.

Set up an AWS EMR cluster by creating a key pair to identify and access the cluster, then save the downloaded key file for future use.

Learn to set up AWS access for a Spark cluster by managing key pairs, using PuTTY and Pageant, and converting PEM keys to PPK with PuTTYgen.

Set up AWS and Spark on EMR, access the management council, sign in with your password and email, and configure the Spark platform for the project.

Learn to set up AWS EMR Spark cluster, configure storage, upload files, and access notebooks in Zeppelin for Spark programming.

Set up an AWS environment for Spark and data analysis, using Zeppelin notebook and SSH access with PPK. Install pandas and essential packages to run Spark analytics in the cloud.

Set up AWS and work with Spark in Python, load a CSV, convert to a data frame, and apply map transformations to reveal results.

Set up the emr cluster and Zeppelin notebook, create a PySpark notebook, learn to configure the Spark context, and run a simple Spark code with cells and basic imports.

Demonstrates creating a Spark RDD from a Python list in Zeppelin, and using collect to reveal RDD contents, while explaining print behavior and RDD addresses.

Explore Python Spark RDD programming on Zeppelin by uploading text files into a packet folder, reading them with Spark RDD transformations and actions, and counting records.

Read a text file into a Python list using Spark RDD on Zeppelin, then use collect to view the contents and count the records with count, finally print the list.

learn rdd programming with spark in zeppelin, using take to extract the top five records and distinguish it from collect for partial or full results.

Explore the map transformation in Spark's RDD programming on Zeppelin. See how map applies a function to each element, not an action, to halve values and print the results.

Apply the map transformation in Spark to split long records into multiple fields using split, converting fields with lambda or defined functions, and extract id, gender, and mass for analysis.

Demonstrate the filter transformation in Python Spark to select only qualified records from an RDD using a lambda in Zeppelin.

Explore how flatMap and map transform RDDs in Spark on Zeppelin to break data into a single document for word frequency analysis, and compare flattened versus nested results.

Learn Python Spark RDD programming in Zeppelin by reading a text file, removing the header, filtering and mapping records, and extracting the first three records for display.

Explore Python Spark RDD programming on Zeppelin by reading a file, removing the header, splitting lines, mapping to the first column (names), applying distinct, and collecting the final records.

Apply a filter transformation using the fifth-column array index to enforce a gender-based condition, yielding only the pop history records that match the gender criterion.

Learn how to use Spark RDD reduce to summarize data by computing a cumulative sum across text file rows, and explore how minus and multiply operations alter the result.

Learn Spark RDD programming in Zeppelin by transforming data with map and reduce by key, reading and cleaning a text file, and computing average income by gender.

Execute a spark rdd word count in Zeppelin, load data, flatMap to words, remove stop words with regex tokenization, then map and reduceByKey to top frequencies over 10.

This lecture introduces Spark data frame basics: read a text file, remove headers, parse lines into rows, define a schema, create a data frame, and query with Spark SQL.

Persist rdd partitions to speed up future actions, avoiding re-reading the text file. Cache serves as a memory-only special case of persist, yielding the same results quickly.

Demonstrates saving a spark rdd to a new folder named story out, performing word count with flatMap and map, and reading back to verify results.

Use Spark accumulators to count records as you read each line, and employ a broadcast variable to apply a shared ratio across all records.

Save a Spark data frame as a parquet file, a compressed format that preserves the data structure and automatically reconstructs the data frame when opened.

Learn how to read data from AWS S3 into a Pandas DataFrame, print the top five records, and convert the DataFrame to a Spark DataFrame for analysis.

Save a pandas data frame to AWS S3 by creating an accessory object and package, buffering contents, and using put to store and verify the data in the cloud.

Learn to set up and manage a Spark cluster with Zeppelin and Jupyter notebook on AWS and Azure, practice PySpark, and monitor and terminate resources to control costs.

Register a Microsoft Azure account and start a free trial, then use the Azure portal to set up a Python, Spark platform and Jupyter Notebook for cloud data science.

Set up an Azure storage account in the dashboard, select storage type and location, create or reuse a resource group, deploy, and pin the resource to the dashboard.

Set up Azure storage containers for blob data, upload files from your local computer, and launch an HD Insight Spark cluster on Linux to access data via a Jupiter notebook.

Set up a spark program in the Azure dashboard using a Jupiter notebook. Read data from the storage system and verify spark functionality with a record count.

Execute the first Python Spark example under Azure by importing packages, setting up the stream, configuring storage and containers, and accessing files via a defined path.

Convert Python lists into a Spark data frame by zipping three lists to form the columns, assigning a schema, and using show to display the resulting data.

Create a spark data frame from a Python list, register it as a temp table, and query it with Spark SQL using select, where, and aggregation to view results.

Learn how to read json files into a Spark dataframe using the read function, leveraging schema inference and comparing json with Parquet as a compressed data format.

Read and write parquet files with spark dataframes and sql, convert between file systems, and filter records by age between 13 and 19.

Learn to treat missing values in spark data frames and pandas data frames by replacing with 999 or 0, creating new data frames, and summarizing with describe and show.

Explore how to use Spark DataFrame group by and aggregation functions to summarize data by age groups, calculate mean income, and retrieve top records with Spark.

Explore spark data frame and SQL aggregation functions, register a table, and filter to show top five ages. Compute the maximum age using aggregation and extract the value from the row.

Learn to define and register user defined functions in Spark to parse email streams, build dataframes with a schema, and use Spark SQL to extract hostnames, lengths, and uppercase emails.

Master applying a Spark UDF to compute utilization (balance over limit) in a dataframe, handling zero denominators, and add a new utilization column with withColumn.

Learn to read data from Microsoft Azure into pandas, create a Spark data frame, add a new age feature, and use group by, show, and collect to explore results.

Explore spark data frame APIs to filter, select, order by age, and limit records; convert the panda state of frame to spark data frame, and compute counts and distinct values.

Learn how to manipulate Spark dataframes with drop duplicates and drop, convert from pandas, and apply the crosstab API to generate cross-tabs between gender and education and perform chi-squared test.

Explore logistic regression under Spark to predict binary outcomes using probability and a threshold for classification, with label point construction, model training, and accuracy evaluation.

Apply tf-idf under spark using pyspark ml feature HashingTF and IDF to transform a sentence into word frequency and tf-idf vectors, inspect word indices, frequencies, and the unique word pack.

Learn how k-means unsupervised clustering segments data using selected variables. Build and evaluate models in Spark with training and test sets, then predict test membership and compare to true labels.

Explore text mining with tf-idf under spark to build a document retrieval system that searches keywords by transforming cleaned, stop-word-filtered text into tf-idf features.

Learn sentiment analysis with Spark on AWS, tokenize text, extract tf-idf features, and build a logistic regression classifier to predict binary sentiment from labeled reviews.

Explain how a credit risk decision tree on spark data frames predicts the probability of a bad loan, using classification or regression and discussing gini impurity.

Perform sentiment analysis with PYENSON spark by loading a text dataset, cleaning text and tokens, and computing tf-idf features in a spark dataframe.

Apply spark to build sentiment analysis pipelines using tf-idf features and logistic regression, with train-test split, feature cleaning, and evaluation via accuracy and auc.

Develop a Python Spark workflow for credit risk analysis in AWS, using a decision tree, with data loading, cleaning, feature engineering, missing-value handling, and label point preparation.

Demonstrate building a credit risk model in PySpark on AWS, using a testing dataset and a decision tree regression to predict probability and evaluate with area under the ROC.

Master the Python Spark exam with 12 questions, including a bonus, complete them independently, then download the question and solution pedia to check your answers.

Use Python to connect to Amazon Redshift, a cloud-based relational data warehouse, run queries to extract and insert data with pandas, and copy data from Esri sources into Redshift.

Connect Python to Redshift, read tables into pandas data frames in a Jupyter notebook, and perform basic analytics and data transfer tasks.

Insert a pandas dataframe into Redshift using psycopg2, including creating and dropping tables and inserting rows via a cursor. Read the data back into a dataframe and review aggregations.

Copy the CSP file into redshift, join it with the customers table in relational database, and use pandas to create a data frame and aggregate the transaction data for analysis.

Demonstrates copying a CSV into Redshift with Python, creating and populating a table, joining datasets, and performing aggregations with Pandas to analyze customer data.