
Explore predictive analytics and how to forecast future outcomes using current data, considering time horizons and validity amid changing factors such as outbreaks and vaccination rates.
Define business success criteria by aligning problem solving with business needs and reducing loan defaulters under 5%. Measure machine learning success by accuracy, performance, and return on investment.
Discover the foundations of data understanding by exploring data types, measurement scales, essential terms, and primary versus secondary data collection techniques.
Explain categorical data and count data within discrete data, contrasted with continuous data, covering binary and multiple categorical data (nominal/ordinal) and interval/ratio scales, with churn and default examples.
Distinguish nominal, ordinal, and interval data using real-world examples from travel, such as flight numbers, gate numbers, and temperatures, and explain subjective versus objective measurements and absolute zero.
Apply design of experiments to data collection by testing marketing promotions, expiry dates, and customer distance to reveal how timing and reach influence coupon redemption.
Introduce crisp data preparation within CRISP-ML(Q), outlining six phases from business and data understanding to data errors, covering objectives, constraints, project charter, and secondary-then-primary data collection.
Compare profits across malaysia and singapore to measure dispersion and variation from the average, using second moment concepts to assess forecast confidence and identify outliers via control charts.
Explore box plots and the differences between percentiles, quantiles, and quartiles, including Q1, Q2 (median), and Q3, and understand how percentiles map to quartiles.
Explore Jupyter and Google Colab to run Python, import pandas, upload and read CSV files, and compare interfaces with Spyder while noting GPUs and TPUs.
This lecture uses scatterplots to reveal direction and strength of relationships (linear or nonlinear), identify outliers and clusters, compare correlation with covariance, outlines data pre-processing techniques and 30-hour training.
Explore data pre-processing fundamentals, including data cleansing, organizing, and typecasting, to convert unstructured logs into structured data and ensure correct data types in Python.
The Data Pre-processing for Data Analytics and Data Science course provides students with a comprehensive understanding of the crucial steps involved in preparing raw data for analysis. Data pre- processing is a fundamental stage in the data science workflow, as it involves transforming, cleaning, and integrating data to ensure its quality and usability for subsequent analysis.
Throughout this course, students will learn various techniques and strategies for handling real-world data, which is often messy, inconsistent, and incomplete. They will gain hands-on experience with popular tools and libraries used for data pre-processing, such as Python and its data manipulation libraries (e.g., Pandas), and explore practical examples to reinforce their learning.
Key topics covered in this course include:
Introduction to Data Pre-processing:
- Understanding the importance of data pre-processing in data analytics and data science
- Overview of the data pre-processing pipeline
- Data Cleaning Techniques:
Identifying and handling missing values:
- Dealing with outliers and noisy data
- Resolving inconsistencies and errors in the data
- Data Transformation:
Feature scaling and normalization:
- Handling categorical variables through encoding techniques
- Dimensionality reduction methods (e.g., Principal Component Analysis)
- Data Integration and Aggregation:
Merging and joining datasets:
- Handling data from multiple sources
- Aggregating data for analysis and visualization
- Handling Text and Time-Series Data:
Text preprocessing techniques (e.g., tokenization, stemming, stop-word removal):
- Time-series data cleaning and feature extraction
- Data Quality Assessment:
Data profiling and exploratory data analysis
- Data quality metrics and assessment techniques
- Best Practices and Tools:
Effective data cleaning and pre- processing strategies:
- Introduction to popular data pre-processing libraries and tools (e.g., Pandas, NumPy)