Complete Python for data science and cloud computing

Discover NumPy and SciPy as the foundation of numerical Python for data analysis and machine learning, and learn how pandas enhances data frames for scientific computing, with Anaconda packaging.

Explore numpy arrays, including one-dimensional and multi-dimensional data, and perform operations such as slicing, indexing, reshaping, stacking, aggregations, and basic mathematical computations.

Create numpy arrays by combining mixed data types and understand how the final dtype is determined, with integers, floats, and strings guiding the result for data analysis.

Demonstrate creating numpy arrays with a specified data type using the empty array constructor, covering integer, float, and string types, and print the results while inspecting the dtype property.

Explore multi-dimensional numpy arrays, build two- and higher-dimensional matrices from lists, inspect shapes, and perform element-wise and concatenation operations for data science workflows.

Learn how numpy array properties differ from functions, and use shape, size, dtype, and itemsize to inspect a two-dimensional array built from a range. Explore transpose via .T.

Slice one-dimensional numpy arrays by specifying a start index, an end index, and an optional step, with end treated as exclusive so the last included index is end-1.

Learn to slice numpy arrays by start and end indices with positive and negative steps, including reverse order, and extract 2d subarrays by selecting specific rows and columns.

Explore fancy indexing in NumPy arrays by selecting rows and columns with index arrays, reshaping ranges into matrices, and reshuffling data to produce ordered or repeated results.

Learn to merge and stack numpy arrays, using concatenate and stack operations along rows or columns, with axis choices and shape considerations for data analysis.

Explore numpy arrays for analysis, including data processing functions, random sampling and loading text files. Learn to copy arrays, fill with a scalar, and reshape and slice data for analysis.

Explore numpy array data processing by converting to lists, updating by index, sorting by axis, computing first and second order differences, and locating nonzero indices and missing values.

Explore Numpy data processing functions, applying repeat to 2d arrays along rows or columns with axis, and using choose and take to index and assemble results.

Explore numpy data processing functions such as A.R.T., search sorted, where, compress, min, max, sum, cumsum, and mean to rank, select, and summarize data across 1D and 2D arrays.

Generate random datasets using uniform and normal distributions, integers, and samples from the random package. Set ranges, sizes, and seeds to reproduce data for analysis and learning.

Load and write data with numpy by reading external text files, specifying file name, delimiter, data type, and optional columns, skip headers, and then save results to a text file.

Learn how to load and save text data with NumPy, read flat files with delimiter handling, select columns, transpose data, and write back as tab- or comma-delimited files.

solve the numpy arrays homework by converting a tuple to a numpy array, identify missing values using non-zero checks, count them, and return a sorted float64 array.

Learn to build numpy arrays from lists, convert ranges, and compose multi dimensional arrays. Explore shape, dimension, transpose, and copying with copy to produce identical arrays.

Explore pandas series objects and their one-dimensional data structure, distinguishing labeled indexing from position-based access. Learn to create series from lists, arrays, or dictionaries for flexible data analysis.

Create Pandas data frames by constructing from arrays, lists, series, dictionaries, or reading data from sources. Define columns and indices, transpose, and inspect shape and values for data frame management.

learn to read external files into a data frame and control headers, missing values, selected columns, index, and data types, with path handling for linux and windows.

Learn to read external files into data frames with csv and text readers, handle missing values, set headers and indices, choose separators, and save results to csv.

Demonstrates reading external files into Pandas data frames using Jupyter notebooks. Use absolute and current directory paths, read csv and flat files, and apply chunked reading for large data.

Learn how to perform arithmetic operations on data frames using scalars, series, and other data frames, with column name matching and handling missing values.

Learn to subset data frames using position and label indexing, apply logical filters (income > 60000, age 40–60), and select specific rows, columns, or scalar values.

Select a single column as a series from a data frame by name with single brackets, and use iloc or loc to slice by index or label.

Explore how to slice a data frame by rows and columns using position-based and label-based indexing, including selecting subsets with conditions like income and gender, illustrated on a coffee dataset.

Show how to slice data frames and series using index values and labels, compare single and double bracket outcomes, and apply boolean and type-based selections to extract numerics and strings.

Learn to manipulate data frames by adding, deleting, and updating rows, using axis to operate on rows or columns, and applying append and concatenate with inner or outer join.

Rename and drop columns or rows in a data frame, with in-place options. Utilize map and apply to update series and data frames, add new features, and insert columns.

Sort and rank data frames with sort_values and sort_index, choosing ascending or descending order and in-place or new frames. Sort by column values or labels, use multiple keys.

Learn how to assign ranking memberships in a data frame with rank, distinguish ranking from sorting, and create ranking groups (quintiles) using cut and qcut.

Learn how to set and manage indices in data frames, including regular and automatic indices, reset and re-index operations, and hierarchical indexing for efficient querying.

Explore pandas indexing methods for data frames, including reset_index, in-place operations, reindexing, and hierarchical index levels, with practical examples using gender and education data.

Learn to reshape data frames by transforming vertical data into horizontal shapes using transpose, pivot, stack, unstack, and melt, with practical city temperature examples.

Learn to reshape data frames by transposing long to wide formats using pivot, stack, and unstack, turning city-season data into columns of temperatures.

Learn to recognize and treat missing and duplicated values in data with Python, assess their impact on regression analyses, and quantify missingness using counts and sums to guide cleaning strategies.

Identify duplicates with value_counts and boolean indexing by id; compare first and last records using groupby to detect differences in other columns, and decide when to drop duplicates cautiously.

Learn to summarize and analyze data with pandas data frames using descriptive statistics: mean, median, count, value_counts, describe, covariance, standard deviation, and variance to reveal patterns and insights.

Use the unique function to list states, then group by state and city and count records to show city totals by state; inspect numerical fields for missing values.

Explore how to create age groups with the cut function, generate a ranking membership variable, and summarize data by state and age group using means of female and male counts.

Convert categorical variables to dummy variables using get_dummies to create 0/1 indicators with a prefix, enabling analysis of attributes like employment and education in a data frame.

Learn to convert categorical columns to binary using dict vectorization, generating sparse or dense arrays and reconstructing data frames for scalable data analysis with many categories.

Learn how sparse matrices efficiently represent large data with mostly zeros, convert categorical variables to dummy variables, and merge with original data for machine learning tasks.

Tackle the first panda's data frame homework with about 50 small questions across three big questions, and download the solution in a Jupiter notebook program for comparison.

Perform year-over-year sales analysis with pandas by transforming dates, filtering for a product, and plotting a bar chart; identify the top five salespersons by total sales.

Learn to use Python with MongoDB for fast data ingestion and analysis, leveraging a document-oriented database with JSON-like structures.

Download and install MongoDB on Windows, set up data and log folders, start the service, verify via command line, and use Python to access MongoDB for data analysis.

Connect Python with MongoDB using PyMongo, set up a localhost:27017 connection, and perform insert, find, and filter operations on the people collection, highlighting key differences from relational databases.

Insert many records into mongo table from a pandas data frame using insert_many, then read back into pandas and apply aggregate to compute averages by gender and sums by education.

Explore data visualization with Matplotlib as a foundational tool, generating bar, pie, and line charts from pandas data, using grouping and mean calculations with titles and legends for clear insights.

Demonstrates data visualization with seaborn to explore a car price dataset using univariate analysis, correlation, and linear regression lines. Visualizes distribution plots, count plots, and swarm plots to reveal relationships.

Install ggplot for python with pip, and explore its one sentence one plot design and layer-by-layer grammar to plot data with variables and add layers via the plus operator.

Install Plotly and learn to generate interactive plots, with offline and online options to host, embed, or download graphs on the Plotly online platform.

Explore offline Plotly to create scatterplots, line graphs, and bar charts using graph objects, traces, and layouts, and generate interactive plots locally.

Demonstrate creating an offline Plotly scatterplot from random data, customize markers with color and size, and configure the layout and zero-based y-axis.

Learn to generate and use an API key for online Plotly to render a scatter visualization, manage credentials and datasets, and display the plot on the Pratley website.

Explore and interpret statistical tests in Python data science, covering distributions, density charts and frequency charts, correlation and linear regression, t-tests, chi-square, nonparametric tests, p-values, confidence intervals, and bootstrapping.

Test whether the population mean equals a fixed value using a one-sample t-test or a normal distribution test, based on normality, and check normality with a histogram and q-q plot.

Explore p-values and t-tests to compare means using one-sample and two-sample designs. Determine when to reject the null hypothesis, based on sample size and variance.

Apply a two-sample t-test to compare spending between test and control groups after a campaign, and interpret the p-value to assess significance.

Introduce analysis of variance (ANOVA) as the extension of the t-test for multiple groups, outlining normality assumptions and how ANOVA compares group means while reducing confounding factors.

Conduct a one-way anova to compare collection rates across four groups and determine whether differences exist, interpreting the f statistic and p-value to identify which group differs.

Explore one-way and two-way ANOVA concepts in Python, assessing education and gender effects on income, with linear regression, interaction terms, dummy variables, and p-values for significance.

Explore how ANOVA and ordinary least squares in Python handle two categorical factors: home ownership and education levels, and interpret interaction effects and p-values to explain spend on food.

Examine repeated measures by comparing before and after results on the same subjects, using paired tests (paired t-test or sign-rank tests), calculating mean differences, their standard deviation, and p-values.

Explore paired tests for data science: use the sign rank sum test for nonnormal paired data and the paired t-test for before–after or multi‑timepoint comparisons, including differences, ranks, and p-values.

Explore how to test relationships between categorical variables using the chi-square test, interpret p-values, and compute expected frequencies from contingency tables.

Perform a chi-square test on two categorical variables, interpret the chi-square statistic and p-value, and examine the contingency table, expected values, and odds ratios with their confidence intervals.

Explore proportion tests by comparing two sample proportions to assess equality using a z-test, normal distribution, and central limit theorem, with examples such as coffee consumption in two groups.

Explore how to perform and visualize statistical tests in python for data science and cloud computing, using anaconda notebooks, including chi-square and one-sample proportion tests, with seaborn plots and p-values.

Apply python-based statistical tests to credit risk data, examining relationships between delinquency and variables such as age and 90-day delinquency counts, with proportion and non-parametric tests.

Using the cloud, this lecture demonstrates a normal distribution with a density function, filters income data, and tests whether the mean is 44000 with a t-test, despite imperfect normality.

Calculate the correlation coefficient and explain it with pocket analysis by grouping income into ranking packets (income_drp) and using a bar chart to show the negative relationship with the target.

Explore simple linear regression with a single predictor x and response y, explaining the intercept and slope, and validating via residuals' normality and independence checks.

Learn how multiple linear regression extends simple regression with many predictors, interpret R-squared and adjusted R-squared, and guard against overfitting using error decompositions and residual analysis.

Explore linear regression with least squares to estimate a, b, and c, compute SST, SSR, and SSD, and evaluate with R square while predicting outcomes from sample data.

Explore feature selection and feature engineering to enhance linear regression, using low-variance filtering, univariate selection, regularization, new feature creation, dummy variables, and interaction terms to improve model performance.

Explore three feature selection methods, variance threshold, univariate chi-square selection, and Lasso, demonstrated on datasets, reducing features from three to two, four to two, and four to three.

Explore feature engineering by creating new features from existing data to boost model performance, including utilization feature from balance and credit limit and the use of dummy variables via binning.

Explore logistic regression by predicting owner versus renter using income and age as predictors from a training dataset, and apply the model to classify new customers.

Explore the logistic regression model for classification using income and age to predict owner probability, apply probability thresholds to classify owners versus renters, and understand the logistic curve.

Estimate logistic regression coefficients from training data using a binary distribution and the likelihood function. Apply supervised learning to predict future data, noting that unsupervised learning lies beyond this section.

Explain logistic regression by modeling the log odds as a linear function of predictors and interpreting coefficients as changes in odds ratios, illustrated with age and income.

Explore logistic regression for binary classification, converting predicted probabilities into owner or renter labels using a threshold, and evaluate with a confusion matrix, sensitivity, specificity, and accuracy.

Explore how to compute sensitivity, specificity, and ROC AUC from logistic regression predictions, interpret cross-tab results, and assess model performance for owner versus renter classification in credit risk contexts.

Explore logistic regression validation through decile analysis and lift, translating predicted probabilities (0–1) into groups to compare group ownership percentages with the overall average.

Explains logistic regression for targeting, demonstrates lift across deciles, and clarifies predictive value. Uses Lorenz curves and sensitivity and specificity curves to assess separation power against random baseline.

Analyze a practical linear regression model to predict weight from workout time and food intake, with normality checks and interpretation of R-squared and p-values.

Explore the use cases of statistical models by assessing a linear regression with p-values, coefficients, and R-squared, using statsmodels to identify significant variables and validate residuals.

Explore linear regression with many variables, focusing on feature selection and correlation to predict customer transaction values. See exploration, missing value checks, and univariate correlation analysis to identify important variables.

Rank correlations to remove the bottom 25 percent and keep the top 300 variables; apply lasso with range-standardized data to select significant predictors, keeping target and id.

Explore how logistic regression uses age and income to predict ownership probability for a binary target, and learn data preparation, model fitting, and probability prediction.

Validate a logistic regression model using predicted scores and ranking groups, generate ranks with qcut, and assess performance by comparing average target and score.

Generate a bar chart of average target and average score across ranking groups using the Tessera table, then compute lift against the overall mean.

Learn to evaluate logistic models with ROC curves and AUC, compute false positive rate and true positive rate using scikit-learn, and understand overfitting risks and KS statistics.

Complete the homework on a linear regression model using the car price.csp data, download the CSP file, follow the question list, and review the lecture for solutions.

Learn how to prepare price data for linear regression by checking normality with plots, truncating long tails, standardizing features, selecting top correlated variables, and evaluating with p-values.

Select the independent variables and the price from the data frame as dependent variable, fit linear regression, and validate the model with p-values, r-squared, mean squared error, and diagnostic plots.

Explore fraud detection using credit card transaction data to predict fraud probability with logistic regression, decision trees, and random forest, emphasizing handling numerous categorical variables.

Explore handling utf-8 encoding, inspecting dtypes, extracting character columns, and building a cross-tab with chi-square tests to assess predictors for fraud in a data science workflow.

Explore univariate analysis for fraud detection by evaluating categorical variables with chi-square statistics and p-values, then assess numeric predictors via logistic regression and AUC performance for feature selection.

Learn to clean fraud data by dropping duplicates on transaction IDs to guarantee unique IDs, apply range standardization (min–max scaling) to numerical features, and combine dummy variables for modeling.

apply chi-square univariate analysis to select 18 variables for the fraud detection project, then use a selector to transform data and identify the chosen feature names.

Predict online product sales using historical product and campaign features to forecast monthly outcomes (m1, m2). The lecture covers dataset structure, missing-values checks, and imputation for independent variables.

Explains Python codes for predicting online product sales by handling missing values with missing-dummy indicators, median imputation, and reshaping data from wide to long using stack and transpose.

Explore Python data prep for predicting online product sales: merge frames, handle missing values, encode categorical variables with label and one-hot encoding, and use sparse matrices for modeling.

Explore feature selection using tree-based importance to reduce 2000 features to key predictors. Build and compare models including random forest regression and gradient boosting regression to predict online product sales.

Explore how to build a credit risk model using logistic regression in Python, including data preprocessing, cross-validation, scorecard construction, and missing-value imputation.

Analyze missing data in credit risk dataset with data slicing, identify variables with missing values—monthly income and number of dependents—compute the missing percentage, and outline imputation or dummy variable options.

The lecture covers building a credit risk scorecard by cleaning data, binning variables to capture nonlinear patterns, and identifying key predictors like utilization, delinquency bands, age groups, and credit lines.

Build a credit risk model with lasso, select non-zero coefficients, use ROC and confusion matrices, and threshold at 0.5 to reach about 93 percent accuracy.

Convert months and year into dummy variables, concatenate them, and apply log transformation to normalize sales data; then build and merge control and testing groups for promotion impact analysis.

Apply linear regression to measure the sales promotion offer's impact, interpret coefficients and p-values for both numeric and categorical variables using statsmodels, revealing a medium positive effect on sales.

Explore how tf-idf vectorization converts text into a sparse feature matrix, selects top features by frequency, and exposes feature names for NLP-based price prediction.

Explore extracting tf-idf features from a document collection to identify important words, build a sparse matrix, and convert results into a data frame for model-ready features.

Build a python data pipeline that generates 60 tf-idf features, creates dummy variables for categorical variables, merges sparse and numerical data, and selects the 300 features most correlated with price.

Learn to build a pricing model and elasticity estimate in Python by selecting items with multiple prices, merging data, applying log transforms, and fitting a linear regression with dummy variables.

Develop a customer and product recommender using a training data set, selecting key products by transaction frequency, classifying customers as purchasers or non-purchasers, and applying kamins segmentation to tailor recommendations.

Explain python codes: customer and product recommender with functions to select initial key products for each segment and a rolling procedure to add more products under a transaction-based threshold.

Learn how Spark and Hadoop integrate for data analysis, using Spark core and RTD in-memory processing with transformations and actions, and leverage PySpark for Python-based data science tasks.

Learn how to register for a free AWS account, complete the signup form, enter payment details, review the usage limits, and log in to get started with AWS services.

Set up an AWS EMR cluster by creating a key pair to identify and access the cluster, then save the downloaded key file for future use.

Learn to set up AWS access for a Spark cluster by managing key pairs, using PuTTY and Pageant, and converting PEM keys to PPK with PuTTYgen.

Learn to set up AWS EMR Spark cluster, configure storage, upload files, and access notebooks in Zeppelin for Spark programming.

Demonstrates creating a Spark RDD from a Python list in Zeppelin, and using collect to reveal RDD contents, while explaining print behavior and RDD addresses.

Explore Python Spark RDD programming on Zeppelin by uploading text files into a packet folder, reading them with Spark RDD transformations and actions, and counting records.

Explore the map transformation in Spark's RDD programming on Zeppelin. See how map applies a function to each element, not an action, to halve values and print the results.

Apply the map transformation in Spark to split long records into multiple fields using split, converting fields with lambda or defined functions, and extract id, gender, and mass for analysis.

Explore how flatMap and map transform RDDs in Spark on Zeppelin to break data into a single document for word frequency analysis, and compare flattened versus nested results.

Apply a filter transformation using the fifth-column array index to enforce a gender-based condition, yielding only the pop history records that match the gender criterion.

Learn how to use Spark RDD reduce to summarize data by computing a cumulative sum across text file rows, and explore how minus and multiply operations alter the result.

Persist rdd partitions to speed up future actions, avoiding re-reading the text file. Cache serves as a memory-only special case of persist, yielding the same results quickly.

Demonstrates saving a spark rdd to a new folder named story out, performing word count with flatMap and map, and reading back to verify results.

Learn how to read data from AWS S3 into a Pandas DataFrame, print the top five records, and convert the DataFrame to a Spark DataFrame for analysis.

Set up an Azure storage account in the dashboard, select storage type and location, create or reuse a resource group, deploy, and pin the resource to the dashboard.

Learn how to read json files into a Spark dataframe using the read function, leveraging schema inference and comparing json with Parquet as a compressed data format.

Read and write parquet files with spark dataframes and sql, convert between file systems, and filter records by age between 13 and 19.

Explore how to use Spark DataFrame group by and aggregation functions to summarize data by age groups, calculate mean income, and retrieve top records with Spark.

Explore spark data frame and SQL aggregation functions, register a table, and filter to show top five ages. Compute the maximum age using aggregation and extract the value from the row.

Learn to define and register user defined functions in Spark to parse email streams, build dataframes with a schema, and use Spark SQL to extract hostnames, lengths, and uppercase emails.

Learn to read data from Microsoft Azure into pandas, create a Spark data frame, add a new age feature, and use group by, show, and collect to explore results.

Explore spark data frame APIs to filter, select, order by age, and limit records; convert the panda state of frame to spark data frame, and compute counts and distinct values.

Learn how to manipulate Spark dataframes with drop duplicates and drop, convert from pandas, and apply the crosstab API to generate cross-tabs between gender and education and perform chi-squared test.

Explore logistic regression under Spark to predict binary outcomes using probability and a threshold for classification, with label point construction, model training, and accuracy evaluation.

Apply tf-idf under spark using pyspark ml feature HashingTF and IDF to transform a sentence into word frequency and tf-idf vectors, inspect word indices, frequencies, and the unique word pack.

Learn how k-means unsupervised clustering segments data using selected variables. Build and evaluate models in Spark with training and test sets, then predict test membership and compare to true labels.

Learn sentiment analysis with Spark on AWS, tokenize text, extract tf-idf features, and build a logistic regression classifier to predict binary sentiment from labeled reviews.

Perform sentiment analysis with PYENSON spark by loading a text dataset, cleaning text and tokens, and computing tf-idf features in a spark dataframe.

Apply spark to build sentiment analysis pipelines using tf-idf features and logistic regression, with train-test split, feature cleaning, and evaluation via accuracy and auc.

Develop a Python Spark workflow for credit risk analysis in AWS, using a decision tree, with data loading, cleaning, feature engineering, missing-value handling, and label point preparation.

Demonstrate building a credit risk model in PySpark on AWS, using a testing dataset and a decision tree regression to predict probability and evaluate with area under the ROC.

Connect Python to Redshift, read tables into pandas data frames in a Jupyter notebook, and perform basic analytics and data transfer tasks.

Insert a pandas dataframe into Redshift using psycopg2, including creating and dropping tables and inserting rows via a cursor. Read the data back into a dataframe and review aggregations.

Copy the CSP file into redshift, join it with the customers table in relational database, and use pandas to create a data frame and aggregate the transaction data for analysis.