GCP - Serverless Computing & AI Platform for Data Science

Explore vertical and horizontal scaling strategies for scalable systems, and see how load balancing and multiple instances enable cloud apps to grow with demand.

Create a Google Cloud Platform free trial account with $300 in credits, start the trial and avoid charges until you enable your account, using a single project for the course.

Install the Google Cloud SDK and set up the gcloud CLI to interact with your cloud account, deploy applications, spin up virtual machines, and automate CI/CD.

Get comfortable with the basics of the gcloud cli by using cloud shell to run core commands, manage project configurations, list projects, and create a cloud storage bucket.

Learn to use batch and gsutil commands in cloud shell to manage files: list, view, create, write, and append; and upload, download, and multithread transfer between local and GCS buckets.

Explore Docker concepts and practical usage, learn to deploy containers to the cloud, and implement continuous deployment with Cloud Run, Cloud Build, and GitHub.

Explore how Docker uses virtualization to package code into portable images and run independent containers. Build images with dependencies, push to a registry, and deploy scalable containers for data science.

Install and start the Docker engine as a background daemon, then verify it runs by launching Docker in the terminal and troubleshoot as needed.

Deploy a simple flask application inside a docker container by creating a dockerfile with python 3.10 slim, copying files, installing dependencies, and running a unicorn server on a mapped port.

Learn to deploy a Python Flask app on Cloud Run, choosing between building and pushing a Docker image to GCR or deploying directly from source code.

Explore Cloud Run scalability parameters—concurrency, min/max instances, platform, and timeout—and how autoscaling, 10-second queues, and 15-minute idling affect performance and cost.

Deploy a python flask app to cloud run via cloud build, loading a US states csv from a GCS bucket into a BigQuery table, using docker and cloud build steps.

learn how to implement continuous deployment with cloud build integrated to a GitHub repo, using a cloud build YAML, creating a trigger, configuring IAM, and automatically redeploying on code changes.

Compare app engine environments by examining standard versus flexible: standard preconfigures runtimes like Python 3.7 for easy deployment, while flexible lets you customize runtimes and supports Docker Rise applications.

Deploy a simple Python Flask application to Google App Engine standard environment using three files (app.yaml, main.py, requirements.txt), configure Python 3.9, and deploy with gcloud to obtain the app's URL.

Split traffic between app engine versions to enable a/b testing, using either ip-based or random routing; configure a 50/50 split with gcloud app services set-traffic and observe versioning.

Deploy a Python Flask app on App Engine that uses a URL parameter to query BigQuery public data from Stack Overflow and return results as JSON.

Learn how caching speeds data retrieval on gcp by storing transient results in memcache to avoid repeated BigQuery queries, and compare shared versus dedicated caches.

Deploy a python Flask app on App Engine that caches BigQuery results with mem cache, stores data in datastore, and renders cached results in HTML page with a one-hour expiry.

Implement a caching workflow in a Python Flask app by passing BigQuery results as keys, caching with memcache, and validating datastore-backed storage to improve subsequent fetches.

Query the chicago taxi trips table in BigQuery for trip_seconds and trip_miles with timestamp. Cache results in memcache or datastore using a unique key and render in an html page.

Deploy a Python Flask app to Google App Engine flexible environment using yaml configuration, Docker image builds, and container registry deployment; leverage runtime config and unicorn entry point.

Deploy a Python application to Cloud Functions that triggers on new csv uploads to a gcs bucket, ingests data into BigQuery, and uses event ark under the hood.

Deploy a simple python http-triggered cloud function using the functions framework, test via an http endpoint, and explore runtimes like Node.js, Go, and Java with gcloud deploy.

Explore a real-time product wishlist use-case in a mobile app, where session id, product id, and wishlist status flow from a microservice to a pubsub topic, cloud functions, and datastore.

Deploy and test the use case code by deploying a cloud function triggered by pub/sub, processing a json payload and persisting session and product wishlist data to datastore.

Understand the machine learning model lifecycle from ideation and data preprocessing to development, deployment on Google App Engine, ML Ops with monitoring, retraining, and prediction serving.

Predict ad clicks with a logistic regression model on Google ad click data, using time spent, age, income, and timestamp; train on a Google Cloud Storage bucket, deploy via endpoint.

Deploy and train a machine learning model on Google App Engine, serialize and store the model in a Google Cloud Storage bucket via an endpoint for retraining and hyperparameter tuning.

Explore a fraud detection use case with anonymized credit card transactions in BigQuery, deploying a random forest model via cyclone on Google App Engine, with data validation and predictions.

Validate a 31-attribute fraud detection dataset in Google App Engine, ensuring all columns exist and the class values are only 0 or 1 before model training.

Learn how GCP workflows orchestrate a two-step serverless pipeline for data validation and model training, defined in a workflow yaml with http get and http post calls and result passing.

Deploy and validate a fraud-detection training model using a workflow and app engine. Leverage a flexible environment, BigQuery data, pandas gbk, and a random forest classifier to generate classification report.

Explore how dataproc serverless uses Spark's dynamic resource allocation to auto-scale executables for PySpark workloads, controlled by properties like dynamic allocation, initial executables, min and max executors, and allocation ratio.

Develop and submit a pyspark job to data proc that reads the public Stack Overflow post table from BigQuery and writes the output to a serverless Spark bucket.

Submit and monitor serverless Dataproc jobs, view batch status in Dataproc, inspect Spark UI and memory usage, and use Spark history for debugging and optimization.

Schedule a daily data proc serverless pyspark job with Airflow using a custom dag that creates, lists, and deletes batches in a serverless environment.

Wrap up by reviewing local patchwork job submission, configuring Spark auto scaling, and triggering serverless Spark jobs via Airflow and Cloud Composer, with cost-conscious resource cleanup.

Explore the Vertex AI UI in GCP, focusing on the dashboard, workbench, pipelines, model development, trainings, model registry, endpoints, and batch predictions for deployment and serving.

Train a custom machine learning model using the web console, uploading dockerfile, training script, and requirements, building and deploying a docker image to run a training job on Vertex AI.

Train a custom model in Vertex AI with the Python SDK to deploy a container and monitor training. Upload the trained artifact to the model registry with a version alias.

Train the model, package artifacts in crc_dir, build a docker image, and deploy to a vertex endpoint for online and batch predictions.

Explore end-to-end model training flow using the Python SDK, including training and deployment scripts, pipeline components, and local endpoint deployment prep.

Upload the two files, install dependencies, build and deploy a local model, then run health checks before pushing to artifact and model registries on Google Cloud AI Platform.

Deploy a model to an endpoint, verify it is ready, and run online or batch predictions with the Python SDK. Explore traffic split, machine types, and replicas for scalable serving.

Explore Kubeflow on Google Cloud as a managed pipelines service for visualizing ml workflows from data collection to deployment, using components and the Dstl compiler.

Explore a Kubeflow and python based lab that walks through queue flow components—from fetch data to train and deploy a fraud model, with outputs written to YAML in GCS.

Build and run a Kubeflow lab pipeline by defining components to fetch data from a CSV into a dataframe, create yaml definitions, and trigger the pipeline while reviewing logs.

Visualize the final Vertex pipeline with three steps completed and a ready endpoint in the chosen region. Explore the pipeline route folder containing get fraud data, train, and deploy artifacts.

Add a model evaluation step to the Kubeflow pipeline before deployment. Demonstrate a random forest classifier with threshold checks (accuracy ≥ 0.8) and ROC/confusion metrics guiding DSTL deployment.

Reuse a json pipeline configuration to train and deploy models, trigger pipelines from a GCS bucket, and integrate with Google Cloud Platform and Google API Python client in Cloud Functions.

Explore the fundamentals of cloud scheduler in Google Cloud, enabling serverless, time-based automation for data science workflows within an AI platform.

Deploy the pops up cloud function, create the pops up topic, then configure the cloud scheduler to invoke the topic on monday at 5:30 p.m. uae time.

Set up an email alerting policy to notify when a cloud function exceeds a max instances value via gcloud deploy, and test by triggering scaling with a csv upload.