End-to-End Machine Learning: From Idea to Implementation

Discover what Git and Github are, why we use them, and quickstart guidance for the hands-on project, with topics you’ll use 90% of the time.

Discover how Git tracks changes across files and enables fast, reliable collaboration with GitHub, including version history, backups, and pull requests.

Install git on Ubuntu using the terminal by updating packages, installing git, and verifying the installation with git --version.

Discover that git is often preinstalled on macOS, then verify with git --version in the terminal and follow the prompt to install if needed.

Create a new GitHub account, verify your email, set a password and username, and enable two-factor authentication using a mobile app such as Google Authenticator, saving recovery codes.

Learn to generate and add an SSH key pair on Linux to link local and remote repositories securely via GitHub, including creating and naming keys and updating settings.

Learn how to generate an SSH key pair on macOS, configure SSH, and add the public key to your GitHub account to secure and synchronize local and remote repositories.

Learn the basic git and GitHub workflow by creating and cloning a repository, staging and committing changes, and pushing local updates to a remote repository.

Learn to revert changes to a tracked file in Git by unstaging and restoring it to the last commit, using git restore --staged and git restore.

Explore git commit history, view progress through descriptive commits, and learn how local and remote repositories synchronize via git push, git log, and origin master.

Learn how to use git aliases to shorten long commands, such as git log --all --graph --decorate --oneline, and configure them globally with git config to streamline your workflow.

Revert to a previous commit with git by inspecting history, using checkout, and optionally creating a new branch, then revert or reset changes to synchronize local and remote histories.

Discover how git diff shows additions and deletions in green and red when comparing local changes to the remote repository using head and origin/master.

Learn branching and merging with git to isolate features and fixes, protect the main branch, and streamline code reviews before merging into master.

Explore merging and pull requests, code review, and how to merge changes into the master branch while managing branches and reviewers.

Use git rebase to squash commits into a clean history and pull teammates’ changes into your branch. Learn interactive rebasing and staying aligned with the remote master during feature development.

learn to stash uncommitted changes in git, switch branches, and later restore them with stash pop or apply, including naming and managing multiple stashes.

Tag commits in git to mark important points, using annotated tags for full objects with messages and optional gpg verification, and lightweight tags as pointers; push with git push --follow-tags.

Learn how to cherry pick a single commit from one branch into another using git, including creating branches, using git cherry-pick, and verifying with git log.

Complete the git and GitHub quickstart, recap key concepts, and move closer to your career goals, while looking forward to the next video.

Provide an introduction to Docker, explain why we use it, install Docker, and cover containers, images, Docker file, persistent data, Docker Compose, and best practices.

Learn how Docker, a software platform, builds, tests, and deploys applications quickly by packaging code into containers with all dependencies; containers simplify delivery and enable scalable deployments.

Install Docker on Linux (Ubuntu or Debian) using the get docker script and curl, then verify with docker run hello world and enable Docker to start on boot.

Install Docker on Windows by downloading Docker Desktop for Windows, completing the setup, restarting, and accepting terms; verify Docker runs a container and explore Docker Hub sharing.

Learn to install Docker on macOS by following Docker's official tutorial, choosing Intel or Apple silicon, and downloading the installer; ask questions in the course Q&A if needed.

Explore Docker containers and how they differ from images, where images hold code and dependencies and containers run as instances, plus Docker Hub as a public repository of base images.

Master docker basics by running containers from ubuntu and pytorch images, using docker run, docker ps, docker exec, docker logs, and container removal with force remove and --rm.

Explore how Docker encapsulates dependencies in a container, enabling multiple Python versions, a clean system, portable images, and safe, scalable updates for your applications.

Explain how Docker images bundle code, libraries, and dependencies in layers, create containers with a transient container layer, and use Docker Hub's official and private images with tags.

Explore how a dockerfile acts as a recipe to build a python 3.8-slim base image, using from, arc, env, run, copy, workdir, and CMD, with docker image build.

Inspect a created docker image and explore environment variables, devmode, and Python 3.8, while learning to optimize Dockerfile layers and caching by stacking commands and backslashes.

Explore how Docker handles container lifetime with immutable, disposable containers and how to preserve important data using persistent data through volumes and bind mounts.

Learn to manage persistent data in Docker using the volume keyword in a Dockerfile, plus named volumes, and mount data with Docker run.

Discover how bind mounting provides persistent data in Docker by mapping a host directory to a container path with Docker run -v, allowing development-time changes to reflect in the container.

Learn how docker compose simplifies running multi-container apps by declaring services, volumes, environment variables, ports, and build options in a yaml file, with examples of Mlflow, Optuna, and Triton.

Learn Dockerfile best practices to keep images lean by using volumes for data, avoiding latest tags, pinning critical package versions, and ordering steps by change frequency.

Discover the core uses of dvc, focusing on data versioning and running pipelines for the project, and get started with practical next steps.

Discover data versioning with DVC alongside Git, including install, track, push to remote storage, and manage dataset versions via DVC metadata.

Learn to access data tracked by DVC from a git workflow, push changes to GitHub, and fetch data using the DVC list and DVC get commands with authentication.

Master end-to-end machine learning pipelines with DVC, using directed acyclic graphs to model data processing, feature extraction, training, and evaluation, while caching results to minimize reruns.

Advance a machine learning pipeline by preparing data, making features, training a logistic regression model, evaluating accuracy, and saving results to YAML, setting up a future discussion of DVC pipelines.

Learn how dvc pipelines manage dependencies and caching across prepare data, make features, train, and evaluate stages, using dvc.yaml, dvc deck, and dvc repro.

Discover metrics and experiments in DVC, though the instructor notes they’re not essential and points you to other tools and DVC tutorials for more.

Explore Hydra, a Python framework by Facebook, for hierarchical configurations from multiple sources and command-line overrides. Install via pip and run multiple jobs with a single command, locally or remotely.

Explore Hydra essentials by building a simple Python app that uses Hydra with Omega Conf, prints configs, and demonstrates command-line overrides for training parameters.

Learn to specify and load configs with Hydra using yaml files, set config path and name, and override or add settings from command line with plus and double plus signs.

Explore OmegaConf in depth, learning to create and load configs, interpolate values, merge configs from files or CLI (including yaml), and understand Hydra as OmegaConf's extension.

Learn to organize experiments with Hydra config groups and per-experiment yaml files, switching models like resnet 18 and resnet 50 using the main config file, command-line arguments, and overrides.

Learn to use a default list to compose configs, set a default experiment like Resnet 18, and override values from the command line; explore the self keyword and config merging.

Use Hydra's multi run to sweep configurations with dash m. Experiment with Resnet 18 and Resnet 50 and loss functions like ark face, cross face, and softmax, using glob syntax.

Hydra automatically sets up logging, routing info level messages to the console and a log file. Create a logger and adjust the level via command line or config.

Learn to debug Hydra config from the command line using dash dash cfg, dash dash job, and the all flag; print configs, inspect Hydra, override values, or target a package.

Discover Hydra's instantiate function to create objects from a config via _target_ keys, with examples from a simple class to a PyTorch optimizer, and use of partial for missing params.

Learn how Hydra packages place config content, override defaults with the defaults list, package directive, or full paths, and see practical examples with task, model, and backbone.

Demonstrate how Hydra configures a PyTorch Lightning training pipeline for an MNIST classifier, illustrating end-to-end instantiation, logging via tensorboard, and effortless experimentation with different backbones and optimizer setups.

Explore the course assignment to train a model on CIFAR-10 and learn how Hydra works, using the task description and course resources.

Explore an end-to-end cifar-10 classification project by building a cifar10 data module and a cifar-10 model with a resnet-50 backbone and a 2048 adapter, with hydra config-driven task switching.

Discover hydro's tab completion you can install to speed command line work on macOS and Linux. Choose Resnet experiments and tweak learning rate, loss, scheduler, model, and optimizer.

Explore structured configs in Hydra, a Python dataclass-based approach to define and validate configuration schemas with runtime and static type checking, supporting basic types, nesting, containers, and optional fields.

Explore structured configs with Hydra by defining a Python data class for an experiment config, registering it in the config store, and setting default model, epochs, and learning rate.

Demonstrate hierarchical static configuration with structured configs, type checking, and integer casting rules, showing how nested data classes form a root config with training and LOS keys.

Learn how to implement config groups with structured configs using Python classes, replacing YAML files, register configs in a main config, and select Resnet 18 or 50 via command line.

Leverage inheritance in structured configs to move common fields to a base class, enforce required attributes with omega conf, and observe resnet models inheriting from the experiment base.

Learn to add a default list to a main config in structured configs using a data class, field, and default_factory, mirroring yaml defaults with a default experiment like ResNet 50.

Use structured configs as a schema to validate YAML configuration files and enforce type checking with Hydra.

Validate config parameters with Pydantic in Hydra workflows, enforcing type checking and constraints. Learn to apply validators for batch size multiples of 32 using pedantic data classes.

Extend a small project with Hydra structured configs to define data module, trainer, and task schemas, including an optimizer, loss function, and model, using setup_config and config store for validation.

Extend the small project with structured configs for CIFAR-10 training and ensure everything works. Follow the course assignment to implement and validate end-to-end machine learning workflows.

Extend a small ml project by adding structured config schemas for cifar-10, updating data module and classification task, and wiring sgd optimizer with Hydra for end-to-end training.

Explore Google Cloud Platform as an optional section to train machine learning models, understanding costs and when to use local resources or rent virtual machines and GPUs.

Learn how to create a Google Cloud Platform account, start a free trial with $300 credits, and set your role and interests in storage, AI, ML, and containerization.

Learn how to create and configure a new project in Google Cloud, decide between single or multiple projects, set permissions, and select the current project via the dashboard.

Install and use gsutil and gcloud commands to configure Google Cloud Platform. Authenticate and set default project, region EU west four, and zone EU west four a for consistent workflows.

Learn to create a Google Cloud Storage bucket with a unique name, choose a single Europe West region to reduce latency, set the standard storage class, and prevent public access.

Learn to manage a Google Cloud Storage bucket by uploading files and folders via the UI or gsutil, using the recursive flag for directories, and downloading results.

Discover how to create and customize a Google Compute Engine virtual machine, choose region and hardware including GPU options, and access via SSH or gcloud to optimize costs.

Learn how Google Cloud quotas limit resources, view gpu limits, and request increases for training. Discover why upgrading to a paid account may be required.

Explore artifact registry on gcp to store and manage docker images and python packages, authenticate, push and pull images, and integrate with compute engine, vms, data processing, and model training.

Explore how GCP firewall rules control traffic to and between VM instances, isolate networks, and monitor activity, using actions, protocols, sources, destinations, and ports to secure a Streamlit demo.

Manage GCP instance groups to run identical virtual machines via an instance template, enabling autohealing, auto scaling, multi-zone deployments, and preemptible or spot machines for distributed training.

Explore Mlflow, open source platform for the end-to-end machine learning lifecycle. It records and compares parameters and results, packages code for reproducibility, and deploys models via a central library-agnostic store.

Explore how mlflow tracking captures parameters, metrics, and artifacts in runs to compare how parameter changes influence results across experiments.

Explore MLflow tracking to run sentiment analysis experiments, log parameters, metrics, and artifacts like confusion matrices, and compare models using a tf-idf and logistic regression pipeline on IMDb data.

Explore MLflow storage types, focusing on backend store and artifact store, and compare local, sqlite, and remote tracking server configurations with cloud storage options like Google Cloud Storage and S3.

Discover how mlflow projects package data science code for reusable, reproducible workflows, detailing project files with name, entry points, dependencies (conda, docker, or system), and run commands.

Create a mlflow project for sentiment classification with a conda.yaml and an entry-point script, then run via mlflow run and track the F1 score.

Discover Mlflow models, a standard packaging format enabling real-time serving via a rest api and batch inference on Apache Spark, with flavors describing models for deployment.

Learn to use mlflow to log models, define signatures, and serve them with sklearn flavor and pyfunc, log artifacts, and run predictions via rest api.

Explore mlflow model registry to version and manage mlflow models with registered models, versions, stages (staging, production, archive), and metadata like annotations and descriptions.

Explore mlflow model registry hands-on, registering the basic sentiment classifier via the UI or Python and managing versions in staging and production. Fetch the registered model with pyfunc and run predictions.

Set up a mlflow tracking server with docker-compose, using a Postgres backend store and a local artifact store. The lecture covers environment files, a docker image, and starting the server.

Explore the production mlflow architecture on gcp, comparing a google compute engine vm tracking server with cloud run, and configure artifact storage on gcs and a serverless postgresql backend.

Create an artifact store on GCP by configuring a cloud storage bucket for the mlflow instance, selecting Europe West four as the region and standard storage with not public access.

Create a serverless Postgres backend store on GCP for mlflow. Configure a mlflow backend user and securely store the password in Secret Manager while preparing deployment scripts.

Learn how to deploy an Mlflow tracking server on Google Compute Engine, using a Postgres database and a GCS bucket as artifact store, with a Docker image and startup script.

log to a deployed mlflow tracking server on a gcloud vm via ssh tunnel to localhost:6100, and verify the playground experiment with metrics and parameters.

Explore how dusk lets you process data that doesn't fit in memory by distributing across workers with a Python-native API similar to pandas.

Explore how Dask dataframes compose many pandas dataframes into a large, parallel dataframe, enabling out-of-core and distributed computations for big data tasks like groupby and joins, using pandas-like API.

Learn to start with dask by converting a pandas dataframe to a dask dataframe, exploring partitions and lazy evaluation, then build parallel task graphs with delayed functions and distributed scheduler.

Learn how to create and store dask dataframes by reading from disk and cloud storage, converting pandas dataframes, and saving as parquet.

Learn best practices for using Dask DataFrame, including partitioning, persist, and parquet workflows to optimize in-memory pandas comparisons.

Explore how shuffling for groupby and join in Dask dataframes can impact performance, and learn strategies to minimize expensive shuffles in larger-than-memory pipelines.

Explore how to use Dask delayed to parallelize custom algorithms, build task graphs, and orchestrate compute calls while following best practices like avoiding mutation and global state.

Parallelize simple computations with thread and process pool executors, then scale using dask distributed with a client, map, scatter, and gather features.

Explore task graphs in Dask, where Python functions produce outputs used as inputs to other tasks, and learn how single-machine and distributed schedulers affect performance.

Deploy a dask cluster via the command line, launch a scheduler and workers, and run distributed computations across multiple machines to scale performance.

Deploy Dask clusters with the Python API by creating a local cluster, connecting a client, and inspecting scheduler, workers, threads, and memory. Prepare for cloud deployment on Google Cloud Platform.

Learn to run distributed model training on GCP using instance groups for single or multi-node setups, with spot machines and SSDs, and deploy via Docker to the GCP artifact registry.

Launch distributed training on GCP using instance groups and the Python API, defining an instance template, creating an instance group from it, and configuring a startup script for all VMs.

Learn to create a Google Cloud instance template with the Python API, defining VM types, boot disks, disks, startup scripts, and metadata to support end-to-end ML workflows.

Learn to create a gcp instance group with the python api, using an instance template creator, setting project id, zone, and name, and list managed instances to obtain ids.

Create startup scripts for instance group deployments, reading instance metadata via curl, setting environment variables, and launching docker-based model training with torch run, followed by automated deletion.

Create and organize configs for GCP VM templates and infrastructure using dataclasses and Hydra, defining boot disk, VM config, metadata, network, and mlflow tracking configurations.

Launch and configure a Google Cloud instance group to deploy a training cluster, using Hydra-driven config, instantiate the group, print job details, and log or view logs and URLs.

Explore tools for distributed model training on GCP and beyond, including instance groups, Vertex AI, Salam, batch jobs, and Kubernetes. Learn practical trade-offs like preemptible machines and disk constraints.

Demonstrates configuring and launching GCP instance groups for ML experiments, adjusting node counts, VM configurations, GPUs, and zones, and reviewing distributed training logs.

Harness fast API, a high performance web framework using Python type hints to power the back end, model predictions, data storage, monitoring, and fast development with automatic interactive documentation.

Set up a virtual environment, install FastAPI and Uvicorn, create main.py with a FastAPI app and a hello world endpoint, then run via Uvicorn --reload and view /docs.

Learn how path parameters work in FastAPI, including type validation, predefined enum values, route order, and using a path converter for file paths.

Learn how to declare query parameters in fast api, using default values and typing.Optional to create optional parameters for items via skip, limit, and path and query parameters.

Define a request body with pydantic models and post a new book to the /books endpoint, including name, author, optional description, and price.