DataScience_Machine Learning - NLP- Python-R-BigData-PySpark

This lecture introduces the role of statistics in data science, highlighting distributions, measures of dispersion, and probabilistic reasoning for machine understanding, while stressing perspective shifts for data professionals.

Explore statistical concepts for loan data, including frequency distributions, central tendency, dispersion, skewness, kurtosis, sampling, data types, hypothesis testing, correlation, regression, supervised, unsupervised, deep learning, and reinforcement learning with Python.

Explore statistical concepts for data analysis, including frequency distributions, central tendency, dispersion, skewness, kurtosis, sampling, hypothesis testing, inference, correlation, regression, machine learning categories, and text weighting.

Explore frequency distribution and the measure of central tendency using histograms and line charts to illustrate normal, skewed, and other distributions with classroom grades and wealth examples.

Explore frequency distributions and the data's central tendency using tables and histograms. Compare mean, median, mode, geometric mean, and harmonic mean with real examples like salaries and horse races.

Explore histograms as representations of frequency distributions, identify ranges from minimum to maximum values, count observations within each bin, and recognize distribution types such as normal and skewed.

Explore central tendency by defining arithmetic mean, median, and mode, and apply geometric and harmonic means to real data, with examples that show when to use each measure.

Explore dispersion by examining range, quartile deviation, mean deviation, and standard deviation, plus the five-number summary. Identify skewness, noting positive and negative skewness and Cortez's concept of distribution shape.

Explore dispersion in data by examining range, quartile deviation, mean deviation, and standard deviation to understand distribution, central tendency, skewness, and interquartile range.

Explore how independent variables, dependent variables, and controlled variables define experiments, using plant height with different inputs like soda, juice, and water, to illustrate measurement and analysis.

Learn how to use samples to represent a population, conduct tests on sample data, and understand simple random sampling (SRS) with and without replacement.

Explore sampling techniques such as simple random sampling with and without replacement, systematic sampling, stratified sampling, and cluster sampling, with examples like vaccination testing and transformer deployments.

Explore how to sample data from a population and learn sampling techniques, including simple random sampling with and without replacement, systematic, stratified, cluster, water, volunteer, and opportunity sampling.

Classify data as quantitative or qualitative and explore discrete and continuous variables, including nominal and ordinal categories, with examples like gender and measurements for machine learning and statistics.

Explore hypothesis testing with null and alternate hypotheses, alpha values, and p-values; learn to compute power and sample size, plan experiments, and decide to reject or accept the null.

identify a clear research question and hypothesis, determine variables and observations, choose a suitable test based on univariate, bivariate, or multivariate analysis, and note confidence intervals.

Learn how to download Python from Python.org, install it on your laptop, and start using it, including how to access the latest version and older releases.

Learn how to set up Python quickly with the Anaconda distribution, installing Python, essential packages, and IDEs like Jupyter and Spyder in a single streamlined installation.

Discover how to use the Anaconda Navigator to launch Jupyter Notebook and Spyder, access preinstalled data science packages, and install new ones from the terminal.

Launch Spyder from Anaconda to write and run Python code in the Spyder editor. Create new untitled files, open existing scripts, and execute Python commands within the environment.

Learn to use Spyder (Python 3.7): view variables in the right consoles, see outputs in the bottom console, and run scripts via the file button or execute snippets with shortcuts.

Python proves valuable by offering libraries for engineering, statistics, and machine learning, enabling Selenium automation and database connectivity, and showcasing interoperability with other languages.

Explore Python's dynamic typing as you create variables, assign values like integers, decimals, and strings, and let Python infer types automatically without explicit declaration, enabling flexible handling in data workflows.

Learn how to create lists with square brackets, access and modify list elements, and perform common operations like append, extend, insert, and remove, highlighting list mutability in Python.

Explore nested data types by creating lists that contain tuples and tuples within lists, and access or update values using nested indexing while noting immutability of certain objects.

Explore Python dictionaries: unique keys, values of any type including lists and nested dicts, and operations to get, access, update, delete, and iterate or sort by keys.

Explore Python sets that maintain unique values, cover union, intersection, and difference with A and B, and manage elements using add, remove, discard, pop, and clear.

Explore Python operators from arithmetic and assignment to comparison, logical, bitwise, identity, and membership operations, with practical examples and incremental value techniques.

Learn how Python uses indentation to define if, elif, else blocks, and master while and for loops, including break, continue, and pass control statements, with practical examples.

Learn how to use Python's input function to capture user input as a string, convert it to int() and perform arithmetic like adding years before printing name and age.

Learn Python file operations: open files in different modes, read lines with readline and readlines, write data, seek and reset pointers, and close files to apply changes.

Explore object oriented programming with Python by using classes as blueprints to create objects, define methods, and call them, illustrated through an employee onboarding example.

Explore Python object oriented programming by examining class and instance scope, public, protected, and private attributes, constructors, getters and setters, and inheritance forms including single, multiple, and multilevel, plus polymorphism.

Explore how to install Python modules in Anakonda, import math, use selective imports and aliases, and apply Banda's, Skateland, Seabourne, and selenium for data engineering, data science, and visualization.

Learn to load diverse datasets with the Parnas library, convert to data frames, and perform basic manipulations, descriptions, filtering, sampling, and joining for data analysis.

Read data from multiple files, inspect a data frame's shape and columns, and perform slice and dice with pandas: select, subset, sort, drop duplicates, and join.

Learn to read data from diverse sources with pandas, apply basic statistical transformations, and sample data with random_state for reproducible mean and median calculations.

Apply basic statistics in Python on data frames, including mean, variance, and standard deviation. Visualize distributions with box plots and scatter plots to explore income, education, and capital gains.

Learn to fix data plots by adjusting the index, converting a column to date format, and using a new column as the x axis to analyze family size across years.

Install and set up R and RStudio, upgrade package versions via CRAN, explore the desktop interface, and manage packages and dependencies to run statistical and machine learning workflows.

Set up and upgrade RStudio for dependencies, install and load packages from CRAN, explore updated functions, and use sample_split to split data into training and testing with a ratio.

Install and upgrade the latest version of R (R 3.6.3) in R Studio, manage CRAN dependencies, and apply changes by configuring global options and restarting the session.

Manage multiple versions of R to handle package dependencies and compatibility, adjust global options, and customize the studio interface to fit your workflow.

Explore how R automates data-type handling with implicit typecasting and no explicit declarations, while enabling variable creation, as.integer conversions, and checks on data frames and joins.

Explore R data types—integer, numeric, logical, character, factor, complex, and date—and learn to declare, convert, and manipulate them with practical examples.

Explore R data structures such as vectors, matrices, arrays, lists, and data frames; learn creating, converting types, indexing, slicing, binding, and basic operations across these structures.

Explore anonymous and named functions in R, including positional and keyword arguments, default values, variadic '...', local and global scope, applying functions to frames with lapply, sapply, and apply.

Explore how to apply functions over dataset objects using apply on rows or columns, compare with for loops, and compute sums and means.

Learn to apply the R apply family to data frames and matrices, using lapply and apply to iterate over objects and compute sums or means across rows and columns.

Learn how to install and load an R library, verify its availability, and use library-based functions (like a normality test) on your data, with detach and attach workflows.

Use the diabetes dataset to apply logistic regression while managing R library dependencies, upgrading from 3.5 to 3.6+, and addressing package availability and 64-bit vs 32-bit requirements.

Learn how to split data into training and testing sets in R using the sample.split function, adjust split ratios, and apply library tools for model validation.

Explore what big data is, its use, and the data lifecycle from transactional systems and data warehouses to RDBMS, emphasizing volume, velocity, and variety for descriptive, diagnostic, and predictive analytics.

Understand how horizontal scaling and distributed computing enable big data processing with Hadoop's master-slave architecture, name nodes, data nodes, and the Hadoop distributed file system.

Explore Hadoop architecture: name node, data nodes, replication, metadata, and heartbeat for fault tolerance and high availability. See how input splits and blocks drive map and reduce jobs across racks.

Hadoop is an open-source framework that enables distributed storage and a Java-based MapReduce core, with tools like Scoop, Flume, Pig Latin, Hive, and Mahout for scalable analytics.

Explore the map reduce architecture in distributed data processing, detailing map and reduce roles, shuffle and data locality. See how combiners, partitioners, speculative execution, and key value inputs optimize performance.

Learn how yarn replaces the old job tracker with a resource manager and application masters, using containers to run mappers and reducers and monitor tasks across the cluster.

Explore Hadoop cluster modes, from standalone/local to fully distributed, detailing how daemons run on dedicated machines, production vs development use, and how resources are managed.

Explore the limitations of MapReduce, including multi-stage jobs, batch-only processing, high data storage needs, and programming complexity, and discover how Spark overcomes these issues as a faster alternative.

Explore Spark as a distributed in-memory processing engine that runs on a cluster, detailing its driver–executor architecture, dag execution, and the role of a cluster manager.

Explore the differences between Spark and Hadoop, and decide when to use in-memory Spark for distributed, fault-tolerant processing, versus disk-based Hadoop, including memory management and streaming versus batch processing.

Explore the Spark framework's core engine, connect to cluster managers, and run in local or distributed modes using Spark core, Spark SQL, streaming, and ML libraries.

Spark offers multiple execution modes where the client communicates with the application master, which spins up the driver and coordinates with the human resource manager to provision executors using containers.

Explore Apache Spark's core abstractions—RDD, data frame, and data set—and learn how transformations and actions run on distributed data using SparkContext, driver, and executors.

Explore a typical big data architecture with storage, Spark processing, batch and real-time analytics, and tools like Cloudera, Hortonworks, and ZooKeeper powering scalable ML workflows.

Learn to install spark on Windows using step-by-step documentation, configure environment variables, and run spark jobs locally with spark-submit and Jupyter notebooks.

Explains Hadoop distribution types, including standalone mode, and guides Windows installation and local practice with Spark using the Hortonworks sandbox on VMware or Oracle VirtualBox.

Install the Hadoop spark sandbox by following Cloudera and Hortonworks documentation, meet RAM requirements, and use Oracle Virtual Box to download, configure, and run the Hortonworks box via online guides.

Learn how Spark transformations and actions enable lazy evaluation, distinguish narrow and wide operations, and practice map, filter, join, and collect on PySpark in the shell.

Create data frames atop data, cache in memory to avoid spills, and use Spark session, Spark SQL, and schema from header for map, group by, and filter transformations.

Apply Spark transformations and actions on RDDs in Jupyter, load raw data from text files, create RDDs and dataframes, and run queries to extract insights.

Explore how to integrate Spark with Hive using the metastore, configure the warehouse location, read and create tables, and perform data manipulation through Spark-Hive integration.

Explore PySpark RDD transformations and actions to perform scalable data processing in big data contexts. Learn core operations that drive efficient data analysis within Spark pipelines.

Apply PySpark data frame operations to read Apple stock data, inspect schema, filter records, select high and low values, and perform aggregations.

Explore the linear regression model: its mathematical formulation, least-squares estimation, and how to identify a best predictive model for continuous data, including assessing fit and verifying core assumptions.

Apply linear regression to predict a continuous variable, such as house price, using multiple independent variables like square footage, bedrooms, area type, and building age.

Explore how least squares identifies the best regression line for linear regression by minimizing the sum of squared errors and using residuals to assess predictions against historical data points.

Identify the least-squares cost function by minimizing squared errors to fit a linear regression line or hyperplane across multi-dimensional data, including polynomial curves when needed.

Fit a linear regression model, evaluate R-squared on test data, and verify key assumptions—linear relationship, zero residuals, multivariate normality, multipolarity checks, outliers via Q-Q plots, autocorrelation, and homoscedasticity.

Analyze the linear regression model using metrics like r-squared, mean squared error, and absolute percentage error. Higher r-squared indicates better fit, while inconsistent values signal issues with variable selection.

Explore how logistic regression models binary outcomes by converting a linear regression into a sigmoid probability between zero and one, using odds and multivariate logistic regression.

Explore how logistic regression uses the sigmoid function to convert predictions into probabilities, choose cutoffs like 0.5, and evaluate models with confusion matrices, sensitivity, specificity, and ROC analysis.

Explore the k nearest neighbors classification algorithm, including selecting odd k values such as 3 or 7, using distance to identify nearest neighbors and predict class by majority vote.