
Personal Introduction
Reproduce the rapid prototyping cycle for covid-19 data across CRISP-DM phases, from data gathering with Johns Hopkins datasets and APIs to modeling with Facebook Prophet and SIR methods.
Discover how data science combines math, statistics, hacking skills, and domain knowhow to extract value from data and enable better business decisions through automation.
Clarify six classes of artificial intelligence, and define terms in data science, machine learning, and statistical modelling, while linking data mining, deep learning, and big data.
Adwise for this section:
Remember the goal of all exercises is to set up everything on your own environment
Python file and Html file show the output of this section
Try to replicate each step in your own environment
Check if the project structure is set up in the correct way
Many different python packages are used throughout this lecture, the requirements file is added to inspect (However, most of the packages should be installed in our own Anaconda environment)
Adwise for this section:
Remember, the goal of the lecture is to set up everything on your own environment to enable you to do more own analysis
Python file and Html file show the output of this section
Try to replicate each step in your own environment
Check if the project structure is set up in the correct way
For the API Access (Lecture 11) you have to organize your own API key
Develop data understanding and collection for covid-19 data by using Johns Hopkins sources, web scraping, REST API services, and pandas in Python notebooks for time-series analysis.
Gather covid-19 data by web scraping the Robert Cole Institute page, extracting the daily state table with BeautifulSoup, and converting it into a data frame.
Wraps up data gathering by comparing three sources: Johns Hopkins, rest API services for US data, and emphasizes automated, version-controlled notebooks, data quality, and pipeline automation for business outcomes.
Learn how to transform a flat table into a relational data model by introducing a primary key, indexing by country and state, and reshaping data into a date-based time dimension.
Explore data understanding by visualizing data frames with pandas, matplotlib, and seaborn; adjust figure size, set axis limits, and plot exponential trends in a logarithmic domain.
Explore creating a dynamic COVID-19 visualization by looping through all countries, updating a multi-country chart with range sliders, styling options, and browser or dash exports.
Create an interactive dashboard using Dash with a country dropdown and callback to update chart traces and layout. Build input-output controls and render a responsive COVID-19 data visualization.
Adwise for this section:
Remember, the goal of the lecture is to set up everything on your own environment to enable you to do more own analysis
Ensure that the packages scikit-learn and scipy are installed
Play with the parameters for regression and filtering to get a feeling for the data set
Try to extract another country which you are interested in and do the analysis
The regression is only an approximation, go with the closed doubling formula to see correct numbers for larger window sizes
Load a selective covid-19 dataset and build helper functions to plot trends with log and linear scales. Align timelines across countries to compare slopes and inspect thresholds.
Explore exponential growth by analyzing slopes and doubling rates (every 2, 10 days) using vectors, broadcasting, and a data frame to compare covid-19 spread in the US and Italy.
Prepare a full walkthrough from business understanding to deployment, building an automated, one-click data pipeline using the John Hopkins dataset to model, process, and compare country data.
Learn how to use group by and apply to derive rolling regression on COVID-19 confirmed cases, using a small test dataset to validate a large relational data workflow.
Apply automated feature transformation to the Johns Hopkins COVID-19 dataset by group-by filtering, computing doubling rate with rolling range regression, and integrating new features into the main data pipeline.
Define data science value for the customer from the start by focusing on measurable business impact. Start with a simple reference model before pursuing more complex data work.
Explore how time series data enable forecasts, from univariate to multivariate models, manage horizon and uncertainty within statistical limits, and apply methods from moving averages to long short term memory modules.
Explore how cross validation helps prevent overfitting in covid-19 data modelling, comparing linear and quadratic models and emphasizing business understanding to select the right model size.
We demonstrate polynomial regression by varying degrees, comparing in-sample and test predictions, and using cross-validation to address overfitting and evaluate metrics like MAE and MAPE.
Explore train-test splits and five-fold cross-validation to stabilize model parameters and prevent overfitting. Compare expanding versus sliding window approaches to protect test data until final evaluation.
Use diagonal plots to diagnose forecast bias and lag, compare growth, linear, and logistic models with capacity, and validate forecasts with cross-validation and rolling windows using Facebook Prophet.
Model the spread of infectious disease with a simple SIR simulation, using susceptible, infected, and recovered groups and beta and gamma driven differential equations to observe their dynamics.
Demonstrate differential equation SIR simulation with 1 million population and 27 infections, using reproduction number of 4 over 50 steps, and fit beta and gamma to data, estimating posterior parameters.
fit a curve to covid-19 data by optimizing differential equation parameters with curve fit and ode integration. explore coupled differential equations and time-series fitting to compare fitted curves with data.
The goal of this lecture is to transport the best practices of data science from the industry while developing a CORONA / COVID-19 analysis prototype
The student should learn the process of modeling (Python) and a methodology to approach a business problem based on daily updated COVID 19 data sets
The final result will be a dynamic dashboard - which can be updated by one click - of COVID-19 data with filtered and calculated data sets like the current Doubling Rate of confirmed cases
Techniques used are REST Services, Python Pandas, scikit-learn, Facebook Prophet, Plotly, Dash, and SIR virus spread simulations + bonus section Tableau for visual analytics
For this, we will follow an industry-standard CRISP process by focusing on the iterative nature of agile development
Business understanding (what is our goal)
Data Understanding (where do we get data and cleaning of data)
Data Preparation (data transformation and visualization)
Modeling (Statistics, Machine Learning, and SIR Simulations on COVID Data)
Deployment (how to deliver results, dynamic dashboards in python and Tableau)