GCP - Google Cloud Professional Machine Learning Engineer

In this lecture, we begin our journey into Linear Regression using BigQuery ML by solving a real-world business problem. The goal of this tutorial is to understand how structured data can be used to train a machine learning regression model directly inside BigQuery, without moving data outside the platform.

We focus on a housing price prediction use case, where the objective is to predict the median house value based on several important input features. This type of problem is common in domains such as real estate, finance, and market analysis, making it an excellent example to understand regression modeling concepts.

Business Problem Overview

The task is to predict the median house value, which acts as the target variable, using the following features:

• Housing median age

• Total number of rooms

• Total number of bedrooms

• Population

• Number of households

• Median income

By analyzing how these features influence house prices, we build a regression model that learns patterns from historical data and makes accurate predictions on unseen data.

Approach and Workflow

In this lecture, we follow a structured and practical workflow that mirrors how machine learning projects are implemented in real environments:

• Uploading raw data files to Google Cloud Storage (GCS)

• Creating a BigQuery dataset and table to store housing data

• Splitting the data into training and testing datasets

• Training a Linear Regression model using BigQuery ML

• Evaluating the model’s performance on test data

• Generating predictions for median house values

Each step is explained clearly so you understand why it is required and how it fits into the overall machine learning pipeline.

Key Concepts Covered

• Understanding regression problems and target variables

• Identifying features and their role in prediction

• End-to-end machine learning workflow in BigQuery ML

• Training, evaluating, and predicting using a regression model

• Applying ML concepts to a real business scenario

In this lecture, we focus on one of the most important foundational steps in any machine learning workflow: preparing and organizing data in BigQuery. Before we can train any machine learning model, the data must be properly stored, structured, and accessible. This tutorial walks you through that complete setup using Google Cloud Console, Cloud Storage, and BigQuery.

You will learn how to move raw data into Google Cloud, create a dataset, and convert a CSV file into a structured BigQuery table that can later be used for machine learning with BigQuery ML.

What You Will Do in This Lecture

This lecture follows a step-by-step, practical approach that mirrors real-world cloud data preparation:

• Accessing the Google Cloud Console

• Creating a Google Cloud Storage (GCS) bucket

• Understanding and selecting bucket configuration options such as:

  • Location type and region

  • Storage class

  • Access control and public access prevention

  • Data protection and encryption settings

• Creating folders inside the bucket to organize machine learning data

• Uploading a CSV file containing housing data to Cloud Storage

Creating a BigQuery Dataset

Once the data is available in Cloud Storage, the lecture moves into BigQuery Studio, where you will:

• Create a new BigQuery dataset

• Select the appropriate region to match the Cloud Storage bucket

• Understand key dataset configuration options, including:

  • External dataset settings

  • Table expiration

  • Encryption options

  • Advanced dataset settings

• Verify dataset creation and explore dataset metadata

This ensures that your data storage and processing locations are aligned, which is a best practice in Google Cloud.

Creating and Exploring a BigQuery Table

After creating the dataset, you will learn how to:

• Create a BigQuery table from a CSV file stored in Cloud Storage

• Configure table settings such as:

  • Table name

  • Schema auto-detection

  • Table type (native table)

• Successfully load the data into BigQuery

You will then explore the created table by:

• Reviewing the table schema (columns, data types, and modes)

• Checking table and storage information

• Previewing the data records directly in BigQuery

• Understanding record counts and table statistics

This step confirms that the data has been loaded correctly and is ready for further analysis.

Key Concepts Covered

• Data ingestion using Google Cloud Storage

• Organizing machine learning data using folders and buckets

• Creating datasets and tables in BigQuery

• Understanding table schema and metadata

• Validating data through table preview and exploration

• Preparing structured data for BigQuery ML workflows

In this lecture, we move to the next critical step in the machine learning workflow: exploring and preparing the data for modeling. Since the dataset and table have already been created in the previous tutorial, this session focuses on understanding the data stored in BigQuery and ensuring it is clean, complete, and ready for training a machine learning model.

Data exploration is a crucial step because the quality of your model depends heavily on the quality of your data. In this tutorial, you will learn how to inspect raw data, analyze data structure, identify data issues, and handle missing values using BigQuery.

Viewing and Understanding the Raw Data

We begin by querying the housing data table to:

• View a sample of records from the dataset

• Limit the number of rows returned for easier inspection

• Understand how the data looks in its raw form

This step helps you become familiar with the dataset before performing any transformations or analysis.

Exploring Schema and Data Types

Next, we examine the table schema to understand how the data is structured:

• Identify all column names in the table

• Review the data type of each column

• Confirm which columns are numerical and which are categorical

Understanding data types is essential before applying machine learning algorithms, as it ensures the data is suitable for analysis and modeling.

Generating Summary Statistics

To gain deeper insights into the numerical data, we calculate summary statistics such as:

• Total number of records in the table

• Minimum, maximum, and average values for key numerical columns

• Distribution insights for features like income and house prices

These statistics help you understand data ranges, spot outliers, and assess whether the values are reasonable for the business problem.

Detecting Missing Values

Missing data can negatively impact machine learning models. In this lecture, you will learn how to:

• Identify columns that contain null or missing values

• Count the number of missing records per column

• Isolate and inspect rows where missing values occur

This step allows you to clearly see which features require data cleaning.

Handling Missing Values with Data Cleaning

After identifying missing values, we focus on cleaning the data by:

• Calculating the mean value of the affected column

• Replacing null values with the column’s mean

• Creating a new cleaned table to preserve the original dataset

This approach ensures that the dataset is complete and suitable for training a regression model without losing important records.

Saving Queries for Reusability

To maintain a clean and reusable workflow, we also save important queries:

• Store data exploration and cleaning queries in BigQuery

• Organize queries for future reference and collaboration

• Prepare the environment for upcoming modeling steps

Key Concepts Covered

• Data exploration using BigQuery

• Understanding schema and data types

• Summary statistics for numerical features

• Identifying and handling missing values

• Creating cleaned datasets for machine learning

• Best practices for data preparation in BigQuery

In this lecture, we move to a critical stage of the machine learning workflow: splitting the prepared dataset into training and testing sets. Since the data has already been explored and cleaned in previous tutorials, this session focuses on creating reliable and non-overlapping datasets that will be used for model training and evaluation.

Proper data splitting is essential to ensure that a machine learning model can be evaluated fairly and performs well on unseen data.

Why Train and Test Splits Matter

Before building a machine learning model, it is important to:

• Train the model on one portion of the data

• Evaluate the model on a separate, unseen portion

• Avoid data leakage and duplicate records

• Ensure consistent and reproducible results

In this lecture, we achieve this using a deterministic hashing strategy inside BigQuery.

Adding a Deterministic Hash Column

We begin by creating a new view that adds a hash-based column to the cleaned housing dataset.

Key ideas covered include:

• Creating a deterministic hash value using selected feature columns

• Converting the hash into a numeric range

• Ensuring the hash value is always non-negative

• Generating a hash bucket between 0 and 99

This hash column ensures that each row is always assigned to the same bucket, making the train-test split consistent across multiple runs.

Ensuring Non-Overlapping Datasets

By using a hash-based approach:

• The same data row will never appear in both training and testing sets

• The split remains stable even if the data is reprocessed

• There is no randomness that could change results between runs

This method is especially useful in production-grade machine learning pipelines.

Creating the Training Dataset

Next, we create a training data view by selecting rows based on the hash bucket range.

Key points include:

• Selecting approximately 80% of the data for training

• Using views instead of physical tables for efficiency

• Verifying the training dataset by querying the view

Creating the Testing Dataset

After creating the training view, we create a testing data view:

• The remaining 20% of the data is assigned to testing

• This dataset is kept separate from training data

• The testing set is used only for model evaluation

This clear separation allows for accurate performance measurement in later steps.

Key Concepts Covered

• Importance of train-test data splitting

• Deterministic hashing for reproducible splits

• Preventing data leakage in machine learning

• Creating views for training and testing datasets

• Validating data splits in BigQuery

In this lecture, we reach one of the most important milestones in the machine learning workflow: building and evaluating a machine learning model. After successfully exploring the data and splitting it into training and testing datasets, this session focuses on creating a Linear Regression model using BigQuery ML and evaluating its performance.

This lecture demonstrates how machine learning models can be trained and evaluated directly inside BigQuery using SQL-based workflows, without exporting data or using external tools.

Creating the Linear Regression Model

We begin by creating a Linear Regression model to predict house prices.

Key concepts covered in this section include:

• Creating a machine learning model inside a BigQuery dataset

• Using a model replacement strategy to ensure consistent results during experimentation

• Selecting linear regression as the model type for predicting numerical values

• Defining the target (label) column as the median house value

• Choosing the appropriate feature columns from the training dataset

Once the model training starts, we review the job execution details and understand how BigQuery ML processes model creation internally.

Understanding Model Training Details

After the model is created, we explore important training-related information, such as:

• Job execution summary and resource usage

• Training iterations and completion status

• Loss values and how they change during training

• Execution stages including preprocessing, training, and evaluation

• Visual graphs showing:

  • Training iteration versus loss

  • Training iteration versus duration

These insights help you understand how the model learns from data and how efficiently the training process is executed.

Exploring Model Metadata and Configuration

Once the model is available, we review its details within BigQuery:

• Model type and model ID

• Dataset and regional location

• Training options and configuration parameters

• Label column and feature columns used during training

• Model modification and expiration settings

This section helps you understand how BigQuery stores and manages machine learning models.

Model Evaluation on Test Data

After training, the model is evaluated using a separate test dataset to measure its real-world performance.

In this part of the lecture, you will learn:

• How to evaluate a trained model using test data

• Why evaluation on unseen data is critical

• How to interpret common regression metrics, including:

  • Mean Absolute Error (MAE)

  • Mean Squared Error (MSE)

  • Mean Squared Log Error

  • Median Absolute Error

  • R-squared (R²) score

  • Explained variance

We compare the evaluation results from the test dataset with the training metrics to ensure the model generalizes well and does not overfit.

Understanding Model Performance

Special attention is given to the R-squared (R²) score, which indicates how well the model explains the variance in house prices. By comparing R² values from both training and test datasets, we assess the consistency and reliability of the model.

Key Concepts Covered

• Creating machine learning models using BigQuery ML

• Linear regression for numerical prediction problems

• Model training workflow inside BigQuery

• Interpreting training metrics and execution graphs

• Evaluating models using test datasets

• Understanding regression performance metrics

• Comparing training and testing results

In this lecture, we complete the end-to-end linear regression workflow by focusing on model inference, also known as making predictions. After successfully creating and evaluating the linear regression model in the previous tutorial, this session demonstrates how to use the trained model to generate predictions on both test data and new, unseen data.

This is a crucial step where machine learning models deliver real business value by producing actionable outputs.

Generating Predictions on Test Data

We begin by making predictions on the test dataset that was created earlier.

In this part of the lecture, you will learn how to:

• Use a trained BigQuery ML model to generate predictions

• Apply the model to test data without including the target variable

• Store prediction results in a new BigQuery table

• Understand the structure of prediction outputs, including:

  • Input feature columns

  • Predicted median house value

This step allows you to compare predicted values with actual values and validate model performance further if required.

Understanding Prediction Output Tables

Once predictions are generated, we explore the newly created prediction table:

• Review the table schema

• Verify feature columns and predicted values

• Preview sample prediction records

This helps confirm that the model is producing outputs in the expected format.

Making Predictions on New, Unseen Data

Next, we move beyond test data and generate predictions on completely new data, simulating a real-world production scenario.

Key steps covered include:

• Uploading new input data to Google Cloud Storage

• Creating a BigQuery table from the new data file

• Ensuring the new dataset contains only feature columns

• Verifying the schema and previewing the data before prediction

This demonstrates how a trained model can be reused to make predictions without retraining.

Storing Predictions for New Data

After preparing the new data, we generate predictions and:

• Save the prediction results in a separate BigQuery table

• Review the schema and preview the prediction results

• Understand how predicted house values are produced for unseen records

This step shows how BigQuery ML supports scalable and repeatable inference workflows.

Key Concepts Covered

• Model inference using BigQuery ML

• Generating predictions on test datasets

• Predicting outcomes for new, unseen data

• Creating and managing prediction tables

• Understanding prediction output structure

• Completing an end-to-end linear regression pipeline

In this lecture, we explore an alternative and more interactive way to execute BigQuery ML workflows by using BigQuery SQL Notebooks. Until now, all SQL queries were executed using the BigQuery SQL Query Editor. In this tutorial, you will learn how to run the same machine learning workflow inside a BigQuery Studio notebook, which provides a more organized and collaborative environment.

This approach is especially useful for experimentation, documentation, and team collaboration.

Introduction to BigQuery SQL Notebooks

We begin by understanding what BigQuery SQL Notebooks are and why they are useful:

• A managed notebook environment within BigQuery Studio

• Supports execution of SQL queries in an interactive, cell-based format

• Enables better organization of machine learning workflows

• Ideal for documenting end-to-end ML pipelines

You will also see the available notebook options and learn how to create a BigQuery SQL notebook.

Enabling Required APIs and Runtime Setup

Before running queries in the notebook, we configure the required environment:

• Enabling the BigQuery Unified API

• Reviewing and confirming permission settings

• Connecting the notebook to a runtime environment

• Selecting the appropriate region for execution

This setup ensures the notebook can securely access BigQuery resources.

Executing SQL Queries in a Notebook Environment

Once the notebook is ready, we demonstrate how to:

• Use special notebook commands to execute BigQuery SQL

• Run SQL queries directly inside notebook cells

• Verify successful execution by querying sample records from tables

This shows how SQL execution in notebooks differs slightly from the standard query editor while producing the same results.

Organizing the End-to-End Machine Learning Workflow

Next, we structure the entire Linear Regression workflow inside the notebook:

• Step 1: Data exploration

• Step 2: Splitting data into training and testing sets

• Step 3: Creating the linear regression model

• Step 4: Evaluating the model

• Step 5: Generating predictions on test data and new data

Each step can be placed in separate notebook cells or grouped logically, making the workflow easier to read, run, and maintain.

Saving and Reusing Notebooks

Finally, we save the notebook so it can be:

• Reused for future experiments

• Shared with team members

• Used as documentation for the complete ML pipeline

This makes BigQuery SQL notebooks a powerful tool for reproducible machine learning workflows.

Key Concepts Covered

• Difference between SQL Query Editor and SQL Notebooks

• Creating and managing BigQuery SQL notebooks

• Runtime configuration and API enablement

• Executing BigQuery SQL inside notebook cells

• Structuring end-to-end ML workflows in notebooks

• Best practices for organizing ML experiments

In this lecture, we begin a new regression use case by introducing Boosted Trees regression in BigQuery ML. Before building the machine learning model, this tutorial focuses on the most important foundation step: data preparation and exploratory data analysis (EDA).

You will work with an insurance dataset and understand how to upload data, create datasets and tables in BigQuery, and explore the data to uncover meaningful patterns that influence insurance charges.

Uploading Insurance Data to Google Cloud Storage

We start by uploading the insurance CSV file to Google Cloud Storage (GCS):

• Use an existing GCS bucket created earlier for BigQuery ML

• Organize data inside the regression folder

• Upload the insurance dataset that will be used for modeling

This step ensures the raw data is securely stored and ready for ingestion into BigQuery.

Creating Dataset and Table in BigQuery

After uploading the file, we move to BigQuery and perform the following steps:

• Create a new dataset named insurance demo

• Select an appropriate region to match the storage location

• Create a native BigQuery table using the CSV file

• Enable schema auto-detection for faster setup

Once the table is created, we review:

• Table schema and column data types

• Feature columns such as age, BMI, smoker status, region, and children

• Target variable charges, which represents insurance cost

Understanding the Regression Problem

This is a regression problem, where the goal is to predict a numerical value:

• Target variable: Insurance charges

• Feature variables: Age, sex, BMI, children, smoker, and region

Understanding this distinction is critical before applying boosted trees regression.

Initial Data Exploration

We begin exploring the dataset to understand its structure and quality:

• View sample records from the table

• Confirm the total number of records

• Validate that the dataset is suitable for regression modeling

Summary Statistics and Data Quality Checks

Next, we analyze key statistics for the target variable:

• Total number of records

• Minimum, maximum, and average insurance charges

• Verification that there are no missing values in the target column

We then extend the missing value analysis across all columns to confirm that the dataset is complete and clean.

Analyzing Categorical Variables

For categorical features, we calculate unique value counts to understand data distribution:

• Sex

• Smoker

• Region

This helps identify how many categories exist and how they may influence the model.

Feature-Level Insights and Relationships

We perform several exploratory analyses to understand relationships between features and insurance charges:

• Smoker vs Charges:
  Clear difference in average insurance charges between smokers and non-smokers

• BMI Categories vs Charges:
  Higher BMI categories show higher average insurance costs

• Age vs Charges:
  Insurance charges increase as age increases

• Children vs Charges:
  No strong correlation observed

• Region vs Charges:
  Moderate variation in charges across regions

• Smoker vs Region:
  Smokers consistently have higher charges across all regions

These insights help justify why boosted trees, which can capture non-linear relationships and feature interactions, are well-suited for this problem.

Key Concepts Covered

• Uploading regression data to Google Cloud Storage

• Creating datasets and tables in BigQuery

• Understanding regression problems and target variables

• Performing exploratory data analysis (EDA)

• Checking data quality and missing values

• Analyzing numerical and categorical features

• Identifying important patterns before model training

In this lecture, we move from data exploration to building a powerful regression model using Boosted Trees in BigQuery ML. After understanding the insurance dataset in the previous tutorial, this session covers the complete machine learning workflow, including data splitting, model training, evaluation, and prediction.

Boosted Trees are well-suited for tabular data and can capture complex, non-linear relationships, making them an excellent choice for predicting insurance charges.

Splitting the Dataset into Training and Testing Sets

We begin by preparing the dataset for modeling using a deterministic, hash-based split.

Key points covered in this section:

• Creating a new table that includes a data split indicator column

• Using a deterministic hashing approach to ensure:

  • Consistent train-test splits

  • No overlap between training and testing data

  • Reproducible results across multiple runs

• Assigning approximately 80% of data to training and 20% to testing

We then verify the split proportions to confirm that the dataset is correctly divided.

Creating Training and Testing Views

To keep the workflow efficient and modular:

• A training data view is created from the split table

• A testing data view is created for model evaluation

• Views allow flexible reuse without duplicating data

This structured setup prepares the data for machine learning model training.

Building a Boosted Trees Regression Model

Next, we create and train a Boosted Trees regression model.

Important concepts explained include:

• Selecting Boosted Tree Regressor for numerical prediction problems

• Defining insurance charges as the target variable

• Choosing feature columns from the training dataset

• Understanding key model configuration options:

  • Number of training iterations

  • Learning rate and its impact on model updates

  • Row subsampling to improve generalization

  • Tree depth to control model complexity

  • Minimum node size to avoid overly specific splits

  • Early stopping to prevent overfitting and unnecessary computation

Each option is explained in simple terms so you understand how it affects model performance.

Understanding Model Training and Performance

After training, we explore the model details:

• Model metadata and training configuration

• Training progress, including:

  • Iteration versus loss

  • Iteration versus duration

• Initial evaluation metrics on training data

Special focus is given to the R-squared (R²) score, which indicates how well the model explains variation in insurance charges.

Evaluating the Model on Test Data

We then evaluate the trained model using the test dataset to measure real-world performance.

Evaluation metrics covered include:

• Mean Absolute Error (MAE)

• Mean Squared Error (MSE)

• Mean Squared Log Error

• Median Absolute Error

• R-squared (R²) score

We compare training and test metrics to ensure the model generalizes well and is not overfitting.

Generating and Storing Predictions

Finally, we use the trained model to:

• Generate predictions on the test dataset

• Store prediction results in a new BigQuery table

• Compare actual insurance charges with predicted values

• Validate that predictions are reasonable and consistent

This step demonstrates how machine learning models deliver real business value through predictions.

Key Concepts Covered

• Deterministic train-test data splitting

• Creating reusable training and testing views

• Boosted Trees regression in BigQuery ML

• Understanding and tuning model parameters

• Evaluating regression models using standard metrics

• Generating and storing predictions

• Building an end-to-end regression pipeline in BigQuery

In this lecture, we begin a new machine learning journey by introducing Binary Classification using BigQuery ML. Unlike regression problems where the goal is to predict numerical values, binary classification focuses on predicting one of two possible outcomes. In this tutorial, we work with an employee attrition dataset to predict whether an employee is likely to stay with the company or leave.

Before building any machine learning model, we focus on data ingestion and exploratory data analysis (EDA), which are critical steps for understanding the problem and preparing the dataset.

Uploading Classification Data to Google Cloud Storage

We start by preparing the raw data:

• Use an existing Google Cloud Storage (GCS) bucket created earlier

• Create a new folder dedicated to classification datasets

• Upload the employee attrition CSV file to Cloud Storage

This ensures that the data is securely stored and ready for processing in BigQuery.

Creating Dataset and Table in BigQuery

After uploading the data, we move to BigQuery Studio and perform the following steps:

• Create a new dataset named attrition demo

• Select the appropriate region to match the Cloud Storage location

• Create a native BigQuery table using the CSV file

• Enable schema auto-detection for faster setup

Once the table is created, we review:

• Table schema and column data types

• Feature columns such as age, gender, department, job role, income, and more

• Target variable attrition, which indicates whether an employee leaves the organization

Understanding the Binary Classification Problem

This is a binary classification problem because:

• The target variable has only two possible values: true and false

• True indicates that an employee leaves the company

• False indicates that an employee stays with the company

Understanding this distinction is essential before applying classification algorithms.

Exploring the Dataset

We then perform exploratory data analysis to understand the structure and quality of the data:

• View sample records from the dataset

• Review column names and data types

• Identify which features are numerical and which are categorical

• Confirm the total number of records

Checking Data Quality and Missing Values

Next, we verify data completeness:

• Count total records in the dataset

• Check for missing values in key columns such as attrition, age, and job role

• Confirm that the dataset does not contain null values in important fields

This step ensures the data is suitable for machine learning.

Analyzing Target Variable Distribution

To understand class balance, we analyze the attrition distribution:

• Count employees who stay versus those who leave

• Identify class imbalance, which is common in real-world classification problems

Understanding class distribution helps in selecting the right evaluation metrics later.

Feature-Level Insights and Relationships

We explore how different features relate to employee attrition:

• Gender vs Attrition: Compare attrition rates between male and female employees

• Department-wise Attrition: Calculate attrition percentage across departments

• Job Role vs Attrition: Identify roles with higher attrition rates

• Numerical Feature Summary: Analyze minimum, maximum, average, and standard deviation for income and other numerical features

These insights provide strong intuition about which features may influence employee attrition.

Key Concepts Covered

• Introduction to binary classification problems

• Uploading classification data to Google Cloud Storage

• Creating datasets and tables in BigQuery

• Understanding target variables and feature columns

• Exploratory data analysis (EDA) for classification

• Analyzing categorical and numerical features

• Understanding class distribution and attrition patterns

In this lecture, we continue our journey into Binary Classification using BigQuery ML by building, evaluating, and using a Logistic Regression model. In the previous tutorial, we prepared the employee attrition dataset and explored the data. In this session, we focus on converting that data into a complete, end-to-end machine learning pipeline.

The goal of this lecture is to predict employee attrition, meaning whether an employee is likely to leave the organization or stay.

Splitting the Dataset into Training and Testing Sets

We begin by preparing the data for model training and evaluation.

Key steps covered include:

• Adding a deterministic hash column to uniquely identify each record

• Creating a new table that includes this hash value

• Ensuring that the same employee record never appears in both training and testing datasets

• Using the hash value to split the data into:

  • 80% training data

  • 20% testing data

This approach guarantees consistent, reproducible, and non-overlapping data splits, which is a best practice in machine learning workflows.

Creating Training and Testing Views

After adding the hash column:

• A training data view is created using records assigned to the training split

• A testing data view is created using records assigned to the testing split

• Views are used to keep the workflow efficient and flexible without duplicating data

This setup prepares clean and structured inputs for model training.

Training a Binary Classification Model Using Logistic Regression

Next, we train a Logistic Regression model, which is one of the most commonly used algorithms for binary classification problems.

In this section, you will learn:

• How to create a machine learning model inside BigQuery

• Why logistic regression is suitable for predicting binary outcomes

• How to define the target variable (attrition)

• How to select feature columns from the training dataset

• How BigQuery ML handles model training automatically

Once training is complete, we explore the model details and configuration.

Understanding Model Training and Metrics

After the model is created, we review important training insights:

• Number of training iterations

• Loss reduction during training

• Learning rate behavior

• Feature columns used in the model

We then analyze training performance metrics, including:

• Accuracy

• Precision

• Recall

• F1 score

• Log loss

• ROC and AUC

You will also see how changing the classification threshold impacts these metrics, which is critical for real-world decision-making.

Evaluating the Model on Test Data

To measure real-world performance, we evaluate the model using the test dataset.

This section explains:

• Why test data evaluation is important

• How training and test accuracy should be compared

• Interpreting evaluation metrics on unseen data

• Verifying that the model generalizes well and is not overfitting

The similarity between training and test accuracy confirms the reliability of the model.

Making and Storing Predictions

Finally, we use the trained model to generate predictions:

• Predict employee attrition on the test dataset

• Generate both:

  • Predicted class (true or false)

  • Predicted probabilities for each class

• Store prediction results in a new BigQuery table

• Understand how to interpret prediction outputs and probabilities

You will also learn how this same process can be applied to new, unseen data in real-world scenarios.

Key Concepts Covered

• Binary classification concepts

• Deterministic train-test data splitting

• Logistic regression in BigQuery ML

• Model training and evaluation metrics

• Precision, recall, accuracy, F1 score, and ROC AUC

• Making predictions with probability scores

• End-to-end classification workflow in BigQuery

In this lecture, we begin a new machine learning topic: Multi-Class Classification using BigQuery ML. Unlike binary classification, where the target variable has only two possible outcomes, multi-class classification involves predicting one class out of many possible categories.

In this tutorial, we work with a wine quality dataset, where the goal is to predict the quality category of wine based on several chemical and physical properties.

Uploading Multi-Class Classification Data to Google Cloud Storage

We start by preparing the raw dataset:

• Use an existing Google Cloud Storage (GCS) bucket created earlier

• Upload a new CSV file containing wine quality data into the classification folder

• Ensure the dataset is securely stored and ready for BigQuery processing

Creating Dataset and Table in BigQuery

After uploading the data, we move to BigQuery Studio and perform the following steps:

• Create a new dataset named wine quality demo

• Select the appropriate region to align with the storage location

• Create a native BigQuery table using the CSV file

• Enable schema auto-detection for faster table creation

Once the table is created, we review:

• Table schema and column data types

• Feature columns such as acidity, alcohol, pH, sugar, and more

• Target variable quality, which contains multiple class labels

Understanding the Multi-Class Classification Problem

This is a multi-class classification problem because:

• The target variable has more than two possible classes

• Wine quality values range across multiple categories

• The model must predict one quality class per record

Understanding the nature of the target variable is critical before building a classification model.

Exploring the Wine Quality Dataset

We then perform exploratory data analysis (EDA) to understand the dataset:

• View sample records from the table

• Confirm total record count

• Identify minimum and maximum values for the quality column

• Understand how many unique classes exist in the target variable

Analyzing Target Class Distribution

To assess class balance, we analyze the distribution of quality classes:

• Count records for each quality category

• Observe whether the dataset is balanced or skewed

• Understand potential challenges for multi-class modeling

Feature-Level Insights

We explore relationships between features and the target variable:

• Analyze average alcohol content by wine quality

• Observe how alcohol levels vary across different quality classes

• Gain intuition about feature importance before model training

Checking Data Quality and Missing Values

Next, we assess data completeness:

• Identify columns with missing values

• Count the number of missing records per column

• Evaluate whether missing values require treatment

Since missing values are minimal compared to the dataset size, we note them for future handling.

Splitting the Dataset into Training and Testing Sets

To prepare the data for modeling, we split it into training and testing datasets using a deterministic approach:

• Add a hash-based data split column to the dataset

• Ensure consistent and reproducible train-test splits

• Assign approximately 80% of records to training and 20% to testing

This approach prevents data leakage and ensures reliable model evaluation.

Creating Training and Testing Views

Finally, we create structured views for modeling:

• A training data view containing only training records

• A testing data view containing only testing records

• Rename columns to remove spaces for better compatibility with machine learning models

This structured setup ensures clean and ready-to-use inputs for the next steps.

Key Concepts Covered

• Introduction to multi-class classification

• Uploading classification data to Google Cloud Storage

• Creating datasets and tables in BigQuery

• Understanding multi-class target variables

• Exploratory data analysis for multi-class problems

• Handling missing values

• Deterministic train-test data splitting

• Preparing clean training and testing views

In this lecture, we move to the most important phase of the multi-class classification workflow: training, evaluating, and interpreting a machine learning model using Boosted Tree Classifier in BigQuery ML. In the previous tutorial, we explored the wine quality dataset and prepared clean training and testing views. In this session, we use that prepared data to build a powerful classification model.

The objective of this lecture is to predict the wine quality category, where the target variable contains multiple classes, making this a true multi-class classification problem.

Training a Boosted Tree Classification Model

We begin by creating and training a Boosted Tree Classifier.

Key concepts explained in this section include:

• Creating or replacing a machine learning model inside a BigQuery dataset

• Using Boosted Tree Classifier, a gradient boosted decision tree algorithm suitable for complex classification problems

• Defining the target variable (wine quality) for multi-class prediction

• Understanding important training options:

  • Maximum number of boosting iterations

  • Learning rate and its impact on model updates

  • Maximum tree depth to control model complexity

  • Row subsampling to reduce variance and prevent overfitting

  • Early stopping to halt training when performance stops improving

Each parameter is explained in simple terms so you understand how it influences model behavior.

Understanding Model Training Execution

Once training starts, we examine the model execution details:

• Job execution summary and resource usage

• Training stages such as validation, preprocessing, training, and evaluation

• Number of planned versus completed training iterations

• Training duration and computational effort

• Execution graphs showing:

  • Iteration versus loss

  • Iteration versus duration

  • Learning rate behavior

These insights help you understand how the model learns during training.

Exploring Model Performance on Training Data

After training, we explore the model details:

• Model metadata and configuration

• Training performance metrics

• Overall training accuracy

• Confusion matrix showing how predictions are distributed across all classes

The confusion matrix helps visualize how well the model distinguishes between different wine quality categories.

Evaluating the Model on Test Data

Next, we evaluate the model using the test dataset, which was kept separate during training.

In this section, you will learn:

• How to evaluate a multi-class classification model on unseen data

• Why test accuracy is important for measuring generalization

• How to compare training accuracy and test accuracy

• Interpreting evaluation metrics for multi-class classification

We observe that test accuracy is slightly lower than training accuracy, which is expected and indicates realistic model behavior.

Making Predictions and Storing Results

After evaluation, we generate predictions on the test data and store the results in a new table.

This part of the lecture covers:

• Generating predicted class labels for each record

• Understanding predicted probabilities for each class

• Selecting the final predicted class based on the highest probability

• Storing prediction results in BigQuery for further analysis

This demonstrates how the trained model can be used for real-world inference.

Understanding Feature Importance

Finally, we analyze feature importance to interpret the model’s decisions.

Key points include:

• Identifying which features contribute most to predictions

• Understanding importance metrics such as:

  • Importance gain

  • Importance weight

  • Importance cover

• Observing which chemical properties most influence wine quality predictions

Feature importance helps explain model behavior and builds trust in machine learning results.

In this lecture, we begin our journey into Time Series Forecasting using BigQuery ML. Time series forecasting is used when data is collected over time and the goal is to predict future values based on historical patterns. This tutorial introduces the core concepts, tools, and workflow required to build a time series forecasting model directly inside BigQuery.

We focus on understanding the ARIMA+ model, which is a powerful and scalable approach provided by BigQuery ML for forecasting time-based data.

Understanding Time Series Forecasting in BigQuery

We start by reviewing the official Google Cloud documentation for time series forecasting to understand:

• What time series forecasting is

• When to use univariate forecasting with ARIMA models

• When to use multivariate forecasting with ARIMA + external regressors

• The overall time series modeling pipeline, including:

  • Data preprocessing

  • Time series decomposition

  • Forecast generation

  • Forecast explanation

  • Anomaly detection

We also discuss how BigQuery ML supports large-scale time series forecasting, including handling multiple time series and automatic scaling using distributed cloud resources.

Uploading Time Series Data to Google Cloud Storage

Next, we prepare the dataset for modeling:

• Use an existing Google Cloud Storage (GCS) bucket

• Create a dedicated folder for time series data

• Upload a CSV file containing historical Superstore sales data

This step ensures that the raw time series data is securely stored and ready for analysis.

Creating Dataset and Table in BigQuery

After uploading the data, we move to BigQuery Studio and:

• Create a new dataset named Superstore Sales Demo

• Select the appropriate region to match the storage location

• Create a native BigQuery table using the uploaded CSV file

• Enable schema auto-detection for faster setup

We then review:

• Table schema and data types

• Key columns such as order date and sales amount

• Total number of records available for forecasting

Exploring and Visualizing Time Series Data

Before building the model, we explore and visualize the data to understand trends and patterns:

• View sample records ordered by date

• Aggregate daily sales values

• Visualize sales trends using:

  • Line charts

  • Bar charts

  • Scatter plots

Visualization helps identify seasonality, trends, and potential anomalies in the data.

Creating and Training a Time Series Forecasting Model

We then build a time series forecasting model using ARIMA+ in BigQuery ML.

Key concepts covered include:

• Selecting ARIMA+ as the model type

• Defining the time column (order date)

• Defining the value column (total sales)

• Adding holiday effects to improve forecast accuracy

• Training the model using aggregated historical sales data

BigQuery ML automatically handles model selection, parameter tuning, and optimization.

Understanding Model Evaluation and Diagnostics

After training, we explore the model details and evaluation results:

• Model metadata and training configuration

• Automatically generated ARIMA model variants

• Key ARIMA parameters such as:

  • Autoregressive terms (p)

  • Differencing (d)

  • Moving average terms (q)

• Detection of:

  • Seasonality patterns

  • Holiday effects

  • Spikes, dips, and structural changes

• Understanding AIC (Akaike Information Criterion):

  • Used as a model quality metric

  • Lower AIC indicates a better-fitting model

This section helps you understand how BigQuery ML evaluates and selects the best forecasting model.

In this lecture, we complete the time series forecasting workflow in BigQuery ML by focusing on model evaluation, future forecasting, forecast explanation, and anomaly detection. In the previous tutorial, we explored and visualized the data and created a time series forecasting model using ARIMA+. This session demonstrates how to use that trained model to extract meaningful insights and actionable predictions.

Evaluating the Time Series Model

We begin by evaluating the trained ARIMA+ model to understand how well it fits the historical data.

In this section, you will learn:

• How to evaluate a time series model using BigQuery ML

• How BigQuery ML generates multiple candidate ARIMA configurations

• How to interpret evaluation outputs visually using charts

• Why evaluation is important before trusting future forecasts

This evaluation step helps confirm that the model has learned the underlying patterns in the data.

Inspecting Model Coefficients

Next, we inspect the ARIMA model coefficients to understand how the model behaves internally.

Key concepts explained include:

• Autoregressive (AR) coefficients

  • Indicate how strongly past values influence current values

  • Moderate positive influence suggests recent history matters

• Moving Average (MA) coefficients

  • Capture the impact of past prediction errors

  • Positive and negative values reflect different correction patterns

• Intercept or drift

  • Represents the baseline trend in the time series

  • Indicates whether the series has an upward or downward trend

This step provides interpretability and helps build confidence in the model.

Forecasting Future Sales

After evaluation, we use the trained model to forecast future sales values.

In this part of the lecture, you will learn:

• How to generate forecasts for a defined future horizon

• Forecasting the next 30 time periods

• Understanding confidence levels in forecasts

• Storing forecast results in a BigQuery table for further analysis

This demonstrates how time series models deliver real business value by predicting future outcomes.

Understanding Forecast Output Columns

We carefully review the forecast output table and explain each column:

• Forecast timestamp: Date for which the prediction is made

• Forecast value: Predicted sales value

• Standard error: Measure of uncertainty in the prediction

• Confidence level: Probability that the true value lies within the interval

• Prediction interval (lower and upper bounds): Expected range of values

• Confidence interval (lower and upper bounds): Statistical confidence range

Understanding these fields helps interpret not just predictions, but also the uncertainty around them.

Explaining the Forecast

We then generate a forecast explanation to understand how the model adjusted historical data:

• Review adjusted time series values

• Analyze confidence and prediction intervals

• Understand how BigQuery ML explains forecast behavior

This step is useful for transparency and model validation.

Detecting Anomalies in Historical Data

Finally, we use the trained model to detect anomalies in historical sales data.

This section covers:

• Identifying unusual spikes or drops in sales

• Understanding anomaly flags (true or false)

• Interpreting anomaly probability scores

• Reviewing lower and upper bounds for expected values

Anomaly detection is extremely useful for identifying outliers, operational issues, or unexpected business events.

Key Concepts Covered

• Evaluating time series models in BigQuery ML

• Interpreting ARIMA model coefficients

• Forecasting future values with confidence intervals

• Explaining time series forecasts

• Detecting anomalies in historical data

• Understanding uncertainty and probability in forecasts

In this lecture, we begin an exciting new topic: Matrix Factorization, which is widely used to build recommendation systems. Recommendation engines power many real-world applications such as movie suggestions, product recommendations, and content personalization. In this tutorial, you will learn how to design and prepare a recommendation system using BigQuery ML.

The focus of this lecture is on data preparation, model setup, and infrastructure requirements needed to train a matrix factorization model in BigQuery.

Understanding Matrix Factorization for Recommendations

Matrix factorization is a machine learning technique used to:

• Learn user preferences based on historical interactions

• Identify relationships between users and items

• Generate personalized recommendations

In this use case, we build a movie recommendation system using user ratings data.

Uploading Recommendation Data to Google Cloud Storage

We begin by preparing the raw datasets:

• Use an existing Google Cloud Storage (GCS) bucket

• Create a dedicated folder for matrix factorization

• Upload two CSV files:

  • Movies (movie details such as title and genres)

  • Ratings (user ratings for movies)

This step ensures the data is securely stored and ready for BigQuery processing.

Creating Dataset and Tables in BigQuery

Next, we move to BigQuery Studio and perform the following steps:

• Create a new dataset named recommendation demo

• Select the appropriate region for processing

• Create two native BigQuery tables:

  • Movies table containing movie metadata

  • Ratings table containing user ratings and timestamps

We then explore both tables to understand their structure and identify the common column (movie ID) that links them.

Creating the Final Training Dataset

To prepare data for matrix factorization:

• We join the movies and ratings tables using the movie ID

• Create a final consolidated table containing:

  • User ID

  • Movie ID

  • Rating

  • Timestamp

  • Movie title

  • Movie genres

This combined dataset forms the foundation for training the recommendation model.

Creating the Matrix Factorization Model

After preparing the final dataset, we configure the Matrix Factorization model:

• Specify the model type as matrix factorization

• Define:

  • User column

  • Item (movie) column

  • Rating column

• Select relevant columns from the final dataset for training

At this stage, we encounter an important limitation related to BigQuery ML infrastructure.

Understanding BigQuery Reservation Requirements

Matrix factorization models cannot be trained using on-demand BigQuery resources. To proceed, we must configure a BigQuery reservation.

In this section, you learn:

• Why reservations are required for matrix factorization

• How to create a reservation using Capacity Management

• Selecting:

  • Reservation type (Enterprise)

  • Maximum slot capacity

  • Autoscaling behavior

• Understanding baseline slots, autoscaling slots, and cost estimation

• Reviewing advanced reservation settings

This step is critical and often overlooked, making it an important real-world lesson.

Assigning the Reservation to the Project

Once the reservation is created:

• We assign it to the active Google Cloud project

• Specify the job type for the reservation

• Ensure BigQuery jobs can use the reserved capacity

Only after this assignment can the matrix factorization model be trained.

Training the Matrix Factorization Model

With the reservation in place:

• The training process starts successfully

• BigQuery ML begins learning latent factors for users and movies

• Training runs using the reserved compute capacity

This completes the setup and training phase of the recommendation system.

Key Concepts Covered

• Introduction to matrix factorization

• Recommendation systems in BigQuery ML

• Preparing user–item interaction data

• Joining multiple datasets for model training

• Creating matrix factorization models

• BigQuery capacity reservations and slot management

• Assigning reservations to projects

• Infrastructure requirements for advanced ML models

In this lecture, we complete the Matrix Factorization workflow by focusing on model verification, recommendation generation, enrichment, and cleanup. In the previous tutorial, we successfully trained a matrix factorization model using BigQuery ML. In this session, we explore the trained model in detail and use it to generate meaningful movie recommendations.

This lecture demonstrates how recommendation systems work end to end and how they can be applied in real-world scenarios.

Verifying the Trained Matrix Factorization Model

We begin by verifying that the matrix factorization model has been created successfully.

In this section, you will learn how to:

• Locate the trained model inside the BigQuery dataset

• Review model metadata such as:

  • Model type and model ID

  • Creation time and location

  • Training configuration and parameters

• Understand important training settings, including:

  • Number of latent factors

  • Iterations completed

  • Regularization settings

  • Early stopping behavior

• Review training diagnostics such as:

  • Iteration versus loss

  • Iteration versus duration

We also examine evaluation metrics like MAE, MSE, and R-squared to understand overall model quality.

Understanding Model Inference for Recommendations

Once the model is verified, we move into inference, where the recommendation system starts delivering value.

You will understand:

• How matrix factorization predicts user–item interactions

• Why there is no label column in recommendation models

• How predicted ratings represent user preference strength

This lays the foundation for generating recommendations.

Generating Recommendations for All Users

Next, we generate recommendations for all users in the dataset:

• Store recommendation results in a new BigQuery table

• Review recommendation outputs including:

  • User ID

  • Movie ID

  • Predicted rating

These predicted ratings indicate how strongly a user is expected to like a particular movie.

Enriching Recommendations with Movie Titles

To make recommendations more meaningful and user-friendly, we enrich them by:

• Joining recommendation results with movie metadata

• Replacing movie IDs with movie titles

• Sorting recommendations by predicted rating

• Reviewing top recommendations across users

You will also learn why predicted ratings may exceed the original rating scale and how they should be interpreted as relative preference scores rather than exact ratings.

Generating Recommendations for a Specific User

In real-world applications, recommendations are often generated per user.

In this section, you learn how to:

• Generate personalized recommendations for a specific user

• Retrieve top recommended movies for that user

• Rank recommendations based on predicted preference

This demonstrates how matrix factorization powers personalized recommendation engines.

Cleaning Up BigQuery Reservations (Important Cost Management Step)

Since matrix factorization requires BigQuery reservations, we perform an important cleanup step:

• Remove reservation assignments from the project

• Delete the reservation entirely

• Ensure no unused resources remain active

This step is critical to avoid unexpected cloud costs and reflects best practices for production environments.

Summary of the Matrix Factorization Workflow

By the end of this lecture, you will clearly understand the complete process:

• Preparing and joining user–item data

• Training a matrix factorization model

• Verifying model training and evaluation

• Generating recommendations for all users

• Creating personalized recommendations

• Managing and cleaning up BigQuery resources

Key Concepts Covered

• Recommendation systems using matrix factorization

• Model inspection and evaluation

• Generating and storing recommendations

• Enriching recommendations with metadata

• Personalized recommendations for individual users

• BigQuery cost and reservation management

• End-to-end recommendation workflow in BigQuery ML

In this lecture, we begin learning about Autoencoders in BigQuery ML, with a strong focus on data preparation and feature engineering, which are critical steps before training any anomaly detection model. Autoencoders are commonly used for identifying unusual patterns or outliers in data, and clean, well-structured input data is essential for their success.

This tutorial prepares the foundation required to build an autoencoder-based anomaly detection system.

Uploading Data to Google Cloud Storage

We start by preparing the raw data:

• Use an existing Google Cloud Storage (GCS) bucket

• Create a new folder dedicated to autoencoders

• Upload the Superstore sales CSV file into this folder

This ensures that the source data is securely stored and ready for BigQuery processing.

Creating Dataset and Table in BigQuery

After uploading the file, we move to BigQuery Studio and perform the following steps:

• Create a new dataset named market demo

• Select the appropriate region to align with the storage location

• Create a native BigQuery table using the uploaded CSV file

• Enable schema auto-detection for quick setup

Once the table is created, we review:

• Table schema and column data types

• Sample records from the dataset

• Total number of records available for analysis

Comprehensive Data Exploration

Before building any machine learning model, we perform a detailed exploration of the data.

In this step, we analyze:

• Total number of records

• Presence of null values in the sales column

• Minimum, maximum, and average sales values

• Standard deviation of sales

This overview helps us understand the overall distribution and scale of the data.

Identifying Obvious Outliers

Next, we check for extreme values in the sales data:

• Analyze the highest sales values

• Review how frequently large values occur

• Identify potential outliers that may impact modeling

This step provides early insights into unusual patterns in the dataset.

Data Cleaning with Additional Feature Engineering

After exploration, we move into data cleaning and feature enhancement, which is essential for anomaly detection.

In this step, we:

• Remove null and invalid sales values

• Ensure sales data is properly converted to numeric format

• Add new analytical features, including:

  • A unique sequential row identifier

  • Sales percentiles to understand relative position in the dataset

  • Z-scores to measure how far each value deviates from the average

These additional features help quantify abnormal behavior in the data.

Building the Features Table for Anomaly Detection

Finally, we prepare the features table, which is required for autoencoder-based anomaly detection in BigQuery ML.

Key ideas covered include:

• Creating a compact feature vector for each record

• Packing the sales value into a structured feature format

• Retaining a row identifier to trace anomalies back to original records

This table represents the final input that will be used by the autoencoder model.

Key Concepts Covered

• Introduction to autoencoders and anomaly detection

• Uploading data to Google Cloud Storage

• Creating datasets and tables in BigQuery

• Exploratory data analysis for anomaly detection

• Identifying extreme values and outliers

• Data cleaning and validation

• Feature engineering using percentiles and Z-scores

• Preparing feature vectors for BigQuery ML models

In this lecture, we complete the Autoencoder-based anomaly detection workflow using BigQuery ML. In the previous tutorial, we explored the data, cleaned it with additional features, and prepared a features table. In this session, we move from data preparation to model training, evaluation, anomaly detection, and operational monitoring.

This lecture demonstrates how autoencoders can be used to identify unusual patterns in large datasets and how to convert those results into a production-ready anomaly monitoring table.

Verifying Cleaned and Feature-Engineered Data

We begin by verifying the cleaned dataset created earlier:

• Review columns such as sales, row ID, sales percentile, and Z-score

• Confirm that the data is properly structured and ready for modeling

• Validate that feature engineering steps were applied correctly

This ensures the autoencoder receives clean and meaningful input.

Training the Autoencoder Model

Next, we train an Autoencoder model for anomaly detection.

Key concepts covered include:

• Creating and training an autoencoder in BigQuery ML

• Limiting training iterations for faster convergence

• Using early stopping to automatically halt training when improvements stop

• Setting minimum relative improvement thresholds to avoid unnecessary computation

We review job execution details such as training duration, data processed, and iteration behavior.

Evaluating the Autoencoder Model

After training, we evaluate the model to understand reconstruction performance:

• Review evaluation metrics such as:

  • Mean Absolute Error (MAE)

  • Mean Squared Error (MSE)

  • Mean Squared Log Error

• Understand how these metrics indicate how well the model reconstructs normal data

• Use both model details and explicit evaluation queries for validation

This confirms that the autoencoder is working as expected.

Quick Anomaly Preview

We then generate a quick anomaly preview to identify the most unusual records:

• Detect anomalies based on reconstruction error

• Specify an approximate contamination rate (percentage of anomalies)

• Sort records by mean squared error to surface the most extreme anomalies

This gives an immediate view of high-risk data points.

Anomaly Summary Statistics

To better understand anomaly distribution, we compute summary statistics:

• Count of normal vs anomalous records

• Percentage of anomalies relative to total data

• Verification that anomaly proportion aligns with the chosen contamination rate

This step provides confidence in anomaly labeling consistency.

Materializing Anomalies into a Monitoring Table

Next, we convert anomaly detection results into a production-ready monitoring table.

In this step, we:

• Create a consolidated table that stores anomaly detection results

• Include key fields such as:

  • Sales values

  • Percentiles and Z-scores

  • Reconstruction error

  • Anomaly flag

• Categorize anomalies by:

  • Anomaly type (high value, low value, mid-range, normal)

  • Severity level (severe, high, medium, low, normal)

• Add timestamps for tracking and auditing

This table is ideal for dashboards, alerts, and downstream analytics.

Querying and Analyzing Anomalies

Finally, we analyze the anomaly monitoring table to extract insights:

• Count anomalies by type

• Analyze severity level distribution

• Combine anomaly type and severity for deeper analysis

• Retrieve the top most extreme anomalies for investigation

These queries demonstrate how anomaly detection results can be operationalized.

Key Concepts Covered

• Autoencoder training in BigQuery ML

• Model evaluation for anomaly detection

• Detecting anomalies using reconstruction error

• Contamination rate and anomaly thresholds

• Creating anomaly monitoring tables

• Classifying anomalies by type and severity

• Querying and analyzing anomaly results

• Building dashboards and alert-ready datasets

In this lecture, we introduce a complete Feature Engineering Pipeline using BigQuery, which is a critical step before building any machine learning model. Feature engineering transforms raw data into meaningful inputs that machine learning algorithms can understand and learn from effectively.

This tutorial focuses on designing, applying, and validating feature engineering techniques using SQL in BigQuery, following a structured and practical approach.

Creating the Dataset and Sample Data

We begin by creating a dedicated dataset named Feature Demo, which will be used throughout this pipeline.

Next, we generate sample customer data to demonstrate feature engineering techniques. This synthetic dataset is intentionally designed to include:

• Numerical features

• Categorical features

• Date-based features

• Target variable for prediction

• Missing (null) values to simulate real-world data issues

The customer data includes fields such as:

• Age

• Gender

• Region

• Last activity score

• Purchase indicator (target variable)

• Customer tenure (date since signup)

• Total spending

• Last visit date

• Support tickets

This dataset helps us demonstrate how raw, imperfect data can be transformed into model-ready features.

Performing Data Quality Checks

Before applying feature engineering, we perform data quality checks to understand the dataset:

• Total number of records

• Count of non-null values in key columns

• Detection of missing values

• Basic statistics such as:

  • Average age

  • Average total spend

  • Purchase rate

These insights help guide decisions such as how to handle missing values and what default values to use.

Comprehensive Feature Engineering Pipeline

This section forms the core of the lecture, where raw data is transformed into rich machine learning features.

1. Handling Missing Values

• Numerical fields are filled with meaningful defaults (based on data statistics)

• Categorical fields are replaced with an “Unknown” category

• Date fields are assigned default placeholder values

• Ensures no null values remain in important features

2. One-Hot Encoding for Categorical Variables

• Converts categorical values (gender, region) into binary indicator columns

• Makes categorical data usable by machine learning algorithms

3. Numerical Transformations

• Calculates time-based features such as:

  • Days since last visit

• Derives behavioral metrics like:

  • Spend per day

4. Interaction Features

• Combines multiple features to capture complex patterns:

  • Age × activity score

  • Activity score × total spend

• Helps the model understand behavioral relationships

5. Feature Normalization

• Applies min–max scaling to numerical features

• Scales values between 0 and 1

• Prevents features with large values from dominating model training

6. Creating Categorical Buckets

• Groups continuous values into meaningful categories:

  • Age groups (young, middle, senior)

  • Activity levels (low, medium, high)

  • Spending tiers (no spend, low, medium, high)

All engineered features are stored in a new table called Customer Features.

Exploring and Understanding the Engineered Features

We then explore the newly created feature table to understand:

• How missing values have been handled

• How categorical variables are encoded

• How normalized values fall within the expected range

• How interaction and bucketed features add context

This step helps validate that feature transformations are logical and useful.

Verifying Feature Engineering Quality

To ensure the pipeline is reliable, we perform validation checks:

• Confirm no null values remain in critical columns

• Verify the target variable is complete

• Validate row counts before and after transformation

This ensures the data is clean, consistent, and model-ready.

Key Concepts Covered

• Importance of feature engineering in machine learning

• Creating synthetic datasets for experimentation

• Data quality assessment

• Handling missing values

• One-hot encoding categorical features

• Feature normalization

• Feature interactions

• Bucketizing continuous variables

• Validating feature pipelines in BigQuery

In this lecture, we continue our Feature Engineering Pipeline in BigQuery by moving from data preparation to machine learning model creation, evaluation, interpretation, and prediction. In the previous tutorial, we completed data creation, quality checks, and comprehensive feature engineering. Now, we use those engineered features to build and compare multiple machine learning models.

This lecture demonstrates how different algorithms perform on the same feature set and how to interpret their results.

Creating a Logistic Regression Model

We begin by training a Logistic Regression model to solve a binary classification problem—predicting whether a customer will make a purchase.

Key concepts covered:

• Why logistic regression is suitable for binary classification

• Defining the target variable (purchase or no purchase)

• Automatically handling class imbalance using class weights

• Limiting training iterations to prevent overfitting

• Using both raw and engineered features for training

After training, we explore the model details, including training behavior, loss curves, and evaluation metrics.

Understanding Logistic Regression Evaluation

We review the evaluation results of the logistic regression model, including:

• Accuracy

• Precision and recall

• F1 score

• Log loss

• Area Under the ROC Curve (AUC)

• Confusion matrix and ROC curves

These metrics help assess how well the model predicts customer purchase behavior.

Training a Random Forest Classification Model

Next, we train a Random Forest classifier, which is more powerful for capturing:

• Non-linear relationships

• Feature interactions

• Complex decision boundaries

Important concepts explained:

• Using multiple decision trees for better generalization

• Voting mechanism across trees

• Enabling global explainability for feature importance analysis

We explore the trained model, review training progress, and understand evaluation outputs.

Comparing Model Performance

We evaluate both models using the same evaluation approach and compare:

• Accuracy

• Precision

• Recall

• F1 score

• Log loss

• AUC

This comparison highlights how different algorithms perform on the same feature-engineered dataset.

Feature Importance Analysis

To understand which features matter the most, we perform feature importance analysis for both models.

Random Forest Feature Importance

• Identifies which features contribute most to prediction decisions

• Measures how much each feature reduces impurity across all trees

• Highlights top drivers such as spending behavior and activity levels

Logistic Regression Feature Weights

• Interprets model coefficients directly

• Positive weights indicate increased probability of purchase

• Negative weights indicate decreased probability

• Absolute weight shows feature impact strength

This section helps you move beyond prediction and into model explainability.

Making Predictions with Probability Scores

Finally, we use the trained model to make predictions:

• Generate predicted labels (purchase or no purchase)

• Output probability scores for each prediction

• Sort customers based on likelihood of purchase

Probability scores allow for more informed business decisions, such as targeting high-probability customers.

Key Concepts Covered

• Training classification models in BigQuery ML

• Logistic regression vs random forest

• Model evaluation and comparison

• Interpreting performance metrics

• Feature importance and model explainability

• Predicting outcomes with probability scores

• Using feature-engineered data for ML models

In this lecture, we complete the Feature Engineering Pipeline in BigQuery by moving beyond basic modeling into advanced customer segmentation, business insights generation, and final model deployment. This session brings together everything we have built so far and shows how feature engineering directly translates into real business value.

By the end of this lecture, you will understand how to transform engineered features into actionable customer intelligence and how to build a production-ready machine learning model.

Recap of Completed Pipeline Steps

Before moving forward, we briefly recap the steps already completed:

1. Creating sample customer data

2. Performing data quality checks

3. Applying comprehensive feature engineering

4. Verifying engineered features

5. Training machine learning models (Logistic Regression and Random Forest)

6. Evaluating model performance

7. Performing feature importance analysis

8. Making predictions with probability scores

With this foundation in place, we move to the advanced stages of the pipeline.

Advanced Customer Segmentation (Step 9)

In this section, we build sophisticated customer segments using behavioral and financial features.

Customer Value Segmentation

Customers are grouped based on spending behavior and activity level into categories such as:

• High Value – Active

• High Value – Inactive

• Low Value – Active

• Low Value – Inactive

This segmentation helps identify customers who contribute the most value and those who need re-engagement.

Churn Risk Assessment

We assess churn risk using:

• Number of support tickets

• Recency of customer visits

Customers are classified into High, Medium, or Low churn risk, enabling proactive retention strategies.

Percentile Rankings

We calculate percentile rankings for:

• Spending behavior

• Activity levels

This shows where each customer stands relative to others on a 0–1 scale.

All these insights are stored in a new Customer Segments table.

Generating Business Insights (Step 10)

Next, we derive actionable business insights from customer segments.

Customer Segment Distribution

We analyze each segment to understand:

• Customer count

• Purchase conversion rate

• Average spending

• Average activity level

This helps identify the most valuable and responsive customer groups.

Churn Risk Analysis

We study how churn risk impacts purchase behavior by analyzing:

• Purchase rate by churn risk

• Average support tickets

• Average days since last visit

This highlights the relationship between customer engagement and churn.

Regional Performance Analysis

We evaluate performance across regions by comparing:

• Customer count

• Purchase rate

• Average spending

These insights help tailor regional marketing strategies.

Final Model Creation with Enhanced Features (Step 11)

With enriched features and segments, we build a final advanced machine learning model:

• Uses a Boosted Tree Classifier

• Trained on:

  • Core demographic features

  • Behavioral features

  • Engineered features

  • Customer segments and risk indicators

• Optimized using:

  • Higher iterations

  • Learning rate tuning

  • Subsampling

• Enabled global explainability for transparency

This becomes the most powerful and business-aware model in the pipeline.

Final Model Comparison

We compare all trained models:

• Logistic Regression

• Random Forest

• Enhanced Boosted Tree

Models are evaluated using:

• Accuracy

• Precision

• Recall

• F1 Score

• ROC AUC

This comparison helps identify the best-performing model based on objective metrics.

Final Predictions with Business Context

We then generate business-ready predictions and store them in a dedicated table.

Each prediction includes:

• Customer details

• Behavioral and financial metrics

• Customer segment

• Marketing priority (High / Medium / Low)

• Purchase probability

• Predicted purchase outcome

• Prediction timestamp

These enriched predictions can be directly used by marketing, sales, and business teams.

Business Insights from Predictions

Finally, we analyze saved predictions to answer real business questions:

• High-priority customer summary

• Marketing priority distribution

• Customer segment vs marketing priority analysis

This shows how machine learning outputs can be transformed into decision-making insights.

Key Concepts Covered

• Advanced customer segmentation

• Churn risk assessment

• Percentile-based customer ranking

• Business insight generation

• Advanced boosted tree modeling

• Model comparison and selection

• Business-context-aware predictions

• End-to-end feature engineering pipeline

In this lecture, we begin our journey into Vertex AI Model Garden, one of the most powerful and important components of Google Cloud Vertex AI. This session focuses on understanding what Model Garden is, why it exists, and how it can be used to quickly access and experiment with modern AI models.

By the end of this lecture, you will have a clear conceptual understanding of Model Garden and how it fits into the broader Vertex AI ecosystem.

What Is Vertex AI Model Garden?

Vertex AI Model Garden is Google Cloud’s centralized hub for AI and machine learning models. You can think of it as an “app store for AI models”, where you can:

• Discover Google’s foundation models

• Explore partner and open-source models

• Experiment with models directly in Vertex AI Studio

• Fine-tune or deploy models for real-world use

It provides a single interface to explore, test, and use a wide variety of pre-trained and open-source models.

Navigating Vertex AI and Model Garden

In this lecture, we walk through:

• Accessing Vertex AI from the Google Cloud Console

• Understanding the main Vertex AI sections such as:

  • Tools

  • Notebooks

  • Vertex AI Studio

  • Agent Builder

  • Data, Model Development, and Deployment

Within the Tools section, we focus on Model Garden and explore its layout and capabilities.

Capabilities and Tasks Supported

Model Garden supports a wide range of AI tasks, including:

• Text generation

• Text classification

• Entity extraction

• Image classification

• Image segmentation

• Text embeddings

• Multimodal use cases

These tasks can be performed using ready-to-use models without building everything from scratch.

Model Collections and Providers

You will learn about different model collections available in Model Garden, such as:

• Google models

• Partner models

• Self-deploy partner models

We also explore models from popular providers, including:

• Google

• Meta

• Anthropic

• Salesforce

• Hugging Face

• Mistral AI

• AI21 Labs

This shows how Model Garden brings together models from multiple ecosystems under one platform.

Foundation Models Overview

The lecture introduces foundation models, including:

• Gemini model family (Gemini 1.x, 2.x, 2.5 variants)

• Image generation and editing models

• Music generation models

• Large language models from partners like Claude, LLaMA, and Mistral

You will understand how these models are grouped and what types of problems they are designed to solve.

Fine-Tunable and Task-Specific Models

We also discuss:

• Fine-tunable models, which can be customized using notebooks or pipelines

• Task-specific solutions, which are pre-built and ready for immediate use

This helps you decide when to use a pre-trained model and when customization is needed.

Exploring Model Details

Using a text generation model as an example, the lecture demonstrates how to:

• View model overview and description

• Understand supported input and output formats

• Review common use cases like summarization and question answering

• Explore prompt design examples

• Access official documentation and references

This section helps you understand how to evaluate a model before using it.

Using Models in Vertex AI Studio

You will see how models can be opened directly in Vertex AI Studio, where you can:

• Interact with models using prompts

• Switch between different models easily

• Adjust output settings and response formats

• Control advanced options like temperature, token limits, and response streaming

This highlights how Model Garden enables fast experimentation without setup overhead.

Key Concepts Covered

• What Vertex AI Model Garden is

• Types of models available

• Model providers and collections

• Foundation vs fine-tunable models

• Using models in Vertex AI Studio

• Model configuration and advanced settings

In this lecture, we take a hands-on approach to building a real-world Generative AI application using Vertex AI Model Garden. You will learn how to transform a simple idea into a fully working web application powered by Google’s Gemini models—without worrying about infrastructure complexity.

The focus of this session is on text generation, prompt design, grounding, and rapid deployment using Vertex AI Studio and Cloud Run.

What This Lecture Covers

In this tutorial, we build a Trip Planner application that generates a detailed, day-by-day travel itinerary based on user inputs. The application is created using a text generation model from Vertex AI Model Garden and deployed as a web app.

Selecting a Model from Model Garden

You will learn how to:

• Navigate to Vertex AI Model Garden

• Choose a text generation task

• Select a suitable Gemini model

• Open the model directly in Vertex AI Studio

• Switch between available Gemini model variants when needed

This step demonstrates how easy it is to start experimenting with powerful foundation models.

Designing the Prompt for the Application

A major part of this lecture focuses on prompt engineering. You will see how to:

• Clearly define the intent of the application

• Specify required input parameters, such as:

  • Starting point

  • Destination

  • Trip duration (in days)

  • User interests (multiple interests supported)

• Describe the expected output format, including:

  • Day-by-day itinerary

  • Activities and experiences

  • Accommodation suggestions

  • Transportation options

  • Estimated costs

You will also learn how Gemini can automatically generate a high-quality prompt based on your intent, saving time and improving consistency.

Enhancing Accuracy with Grounding

To improve reliability and relevance, the lecture demonstrates how to:

• Enable grounding with Google Search

• Anchor model responses to verifiable, real-world information

• Reduce hallucinations in generated content

This is especially important for travel-related applications where accuracy matters.

Deploying the Application as a Web App

Once the prompt is finalized, you will learn how to:

• Convert the prompt into a working web application

• Deploy the app using Cloud Run

• Configure public access for testing

• Understand how Vertex AI automatically handles backend setup

This shows how quickly you can go from prompt to production-ready app.

Testing the Trip Planner Application

You will see a live demonstration of the application by providing inputs such as:

• Starting city

• Destination country

• Trip duration

• Interests like culture, food, art, and adventure

The application generates:

• A complete multi-day itinerary

• Transportation recommendations

• Accommodation options

• Cost estimates for activities, food, and travel

This validates how Generative AI can create rich, structured outputs from simple inputs.

Exploring the Cloud Run Service

The lecture also covers:

• Viewing application metrics such as:

  • Request count

  • Latency

  • CPU and memory usage

• Exploring deployed application files

• Understanding how prompts power the app logic

• Knowing how to update the application by modifying the prompt and redeploying

Cleaning Up Resources

To avoid unnecessary costs, you are shown how to:

• Safely delete the Cloud Run service

• Confirm that resources are fully removed

This highlights best practices for cost management on Google Cloud.

Key Takeaways from This Lecture

By the end of this lecture, you will understand:

• How to use Vertex AI Model Garden for application development

• How to design effective prompts for real-world use cases

• How grounding improves AI reliability

• How to deploy AI-powered applications quickly using Cloud Run

• How Generative AI can be turned into a usable product with minimal effort

In this lecture, you will learn how to build a real-world Entity Extraction application using Vertex AI Model Garden. The goal of this tutorial is to demonstrate how Generative AI models can automatically extract structured information from unstructured job descriptions, including text, PDF files, and images.

This session focuses on Entity Extraction, prompt design, application deployment, and iterative improvement using Vertex AI Studio and Cloud Run.

What You Will Build in This Lecture

You will create an application that:

• Accepts job descriptions as input

• Supports text, PDF, and image formats

• Automatically extracts key entities such as:

  • Job title

  • Company name

  • Location

  • Employment type

  • Required experience

  • Education qualifications

  • Salary range

  • Application deadline

This type of application is commonly used in HR systems, recruitment platforms, and resume screening tools.

Selecting an Entity Extraction Model

You will start by:

• Navigating to Vertex AI Model Garden

• Choosing the Entity Extraction task

• Selecting a suitable Gemini model

• Opening the model in Vertex AI Studio

• Understanding how model versions can change and how to proceed with available variants

This step shows how Model Garden simplifies access to powerful foundation models.

Exploring the Prompt Gallery

Before creating the application prompt, you will explore the Prompt Gallery, which provides ready-made prompt examples for different use cases such as:

• Chatbots

• Reviews analysis

• Document processing

• Text classification

This helps you understand how prompts are structured and how they can be adapted for your own application.

Designing an Effective Entity Extraction Prompt

A key part of this lecture is prompt engineering. You will learn how to:

• Start with a rough or imperfect prompt intent

• Use Gemini’s “Help me write” feature to refine it

• Convert vague requirements into a clear system instruction

• Define exactly which entities should be extracted

• Specify supported input formats such as text, PDF, and images

You will see how refining the prompt dramatically improves extraction accuracy and application behavior.

Deploying the Entity Extraction Application

Once the prompt is finalized, you will:

• Deploy the application directly from Vertex AI Studio

• Enable public access for easy testing

• Understand how the app is automatically hosted using Cloud Run

• Learn how quickly a proof-of-concept AI app can be created

Testing the Application with Real Data

You will test the application using:

• Plain text job descriptions

• PDF job description documents

The application successfully extracts and displays structured entities such as job title, company name, location, experience, and salary information, demonstrating the practical power of Generative AI for document understanding.

Improving the Application Through Iteration

You will also learn an important real-world skill:

• How to redeploy the application when requirements change

• How modifying the prompt can enable new features, such as file uploads

• Why iterative prompt refinement is critical in AI application development

This shows how AI applications evolve through experimentation.

Exploring the Deployed Application

The lecture briefly covers:

• Viewing the application in Cloud Run

• Accessing the source code editor

• Understanding where prompts and configurations are stored

• Navigating between Vertex AI Studio and Cloud Run

Cleaning Up Resources

To follow best practices, you will see how to:

• Delete deployed applications

• Avoid unnecessary cloud costs

• Keep your Google Cloud environment clean

Key Takeaways from This Lecture

By the end of this lecture, you will understand:

• How to build an Entity Extraction application using Vertex AI

• How to design effective prompts for structured data extraction

• How to support multiple input formats like text, PDF, and images

• How to deploy AI applications quickly using Cloud Run

• How to iteratively improve AI behavior through prompt refinement

In this lecture, you will learn how to build a multi-language translation application using Vertex AI Model Garden. The focus of this tutorial is on using Generative AI models to translate product descriptions from English into multiple languages through a simple, deployable application.

This lecture demonstrates how quickly and effectively translation use cases can be implemented using Google Cloud’s Gemini models and Vertex AI Studio.

What You Will Build in This Lecture

You will create a translation application that:

• Accepts product descriptions written in English

• Translates the content into multiple target languages, such as:

  • Hindi

  • Spanish

  • French

  • German (and other supported languages)

• Displays all translated outputs in a clear and structured format

This type of application is commonly used in e-commerce platforms, global marketplaces, and multilingual content systems.

Exploring the Translation Task in Model Garden

You will begin by:

• Navigating to Vertex AI Model Garden

• Selecting the Translation task

• Choosing a Gemini model suitable for language translation

• Opening the model in Vertex AI Studio

This step helps you understand how Model Garden categorizes AI tasks and makes them easy to explore and use.

Designing the Translation Prompt

A major part of this lecture focuses on prompt design. You will learn how to:

• Define a clear prompt intent for translation

• Use Gemini’s “Help me write” feature to automatically generate a high-quality system instruction

• Specify:

  • Source language (English)

  • Multiple target languages

  • The type of content being translated (product descriptions)

• Save and reuse prompts for future applications

This ensures the translations are accurate, consistent, and suitable for production use.

Deploying the Translation Application

Once the prompt is finalized, you will:

• Save the prompt with region and autosave settings

• Use the Build with code option to deploy the application

• Allow public access for easy testing

• Automatically deploy the application using Cloud Run

This shows how quickly an AI-powered web application can be launched with minimal setup.

Testing the Application

After deployment, you will:

• Open the live application

• Enter a product name and description

• Submit the input for translation

• View the translated content in multiple languages

This confirms that the application works as expected and demonstrates real-world usability.

In this lecture, you will learn how to generate high-quality images using Vertex AI Model Garden. The focus of this tutorial is on understanding how foundation image generation models can be used to create visually rich and creative images using natural language prompts.

This session introduces image generation as a practical and powerful use case of Generative AI on Google Cloud.

Introduction to Image Generation in Model Garden

The lecture begins by exploring the Model Garden interface inside Vertex AI and selecting the Image Generation category. You will learn how to:

• Browse available foundation image models

• Understand the difference between model variants

• Select an advanced image generation model suitable for creative tasks

You will work with a high-quality image generation model designed for professional-grade outputs.

Understanding the Image Generation Model

Before generating images, the lecture explains important model information, including:

• Model overview and capabilities

• Common use cases such as illustration, design, and creative artwork

• Available documentation for deeper learning

• Pricing considerations for image generation workloads

This helps you make informed decisions when choosing models for real-world projects.

Configuring Image Generation Settings

You will then open the model in Vertex AI Studio and configure key generation options, such as:

• Aspect ratio (using the default 1:1 format)

• Safety settings, including:

  • Person generation settings

  • Age constraints

  • Safety filter thresholds

• Deployment region selection

These configurations ensure that generated images follow safety guidelines and meet project requirements.

Writing an Effective Image Prompt

A major focus of this lecture is prompt design for image generation. You will learn how to:

• Describe artistic style clearly

• Specify colors, mood, and visual tone

• Define subject behavior and expressions

In this tutorial, the prompt is designed to generate a bright, colorful, cartoon-style illustration with:

• Bold outlines

• Exaggerated features

• Playful expressions

• A cheerful and friendly atmosphere

A subject is added to the prompt to guide the model in creating a specific visual concept.

Generating and Reviewing Images

After submitting the prompt, the model generates multiple image variations. You will:

• Review different generated image options

• Understand how the model interprets your prompt

• See how slight prompt changes affect visual output

This helps you evaluate and refine creative results.

In this lecture, you will learn how to build a multi-modal Generative AI application using Vertex AI Model Garden. Multi-modal models can understand and process both images and text together, enabling powerful use cases such as automatic content creation for social media and video platforms.

This tutorial demonstrates how to use Gemini 2.5 Pro, one of Google’s advanced multi-modal models, to generate marketing and social media content directly from uploaded images.

Introduction to Multi-Modal Generation

The lecture begins by introducing multi-modal generation, where a single AI model works with multiple input types simultaneously. In this case, the model analyzes:

• Images (visual context)

• Text prompts (instructions and intent)

You will see how this capability enables richer and more contextual outputs compared to text-only models.

Selecting a Multi-Modal Model

You will explore the Gemini 2.5 Pro model inside Vertex AI Model Garden and review:

• Model overview and capabilities

• Supported input types (text + image)

• Common use cases such as content generation, media analysis, and creative automation

The model is then opened in Vertex AI Studio for interactive use.

Designing a Multi-Modal Prompt

A key part of this lecture focuses on prompt design for multi-modal tasks. Using the built-in assistance feature, you will create a structured prompt that instructs the model to:

• Analyze an uploaded image

• Generate multiple types of content from that image, including:

  • Instagram Reel title

  • Instagram caption

  • YouTube thumbnail text

  • YouTube video title

  • YouTube video hashtags

The lecture shows how the system can suggest a well-structured prompt and how you can customize it further if needed.

Deploying the Application

Once the prompt is ready, you will deploy the application as a web app powered by Cloud Run. During deployment, you will:

• Allow public access for testing

• Create a fully functional interactive application

• Wait for the deployment to complete and access the live app

This allows you to test the multi-modal experience in a real user interface.

Testing with Real Images

After deployment, you will test the application using different images:

Example 1: Product Image

• Uploading a smartphone image

• Automatically generating:

  • Instagram Reel title and caption

  • YouTube video title

  • Thumbnail text

  • Relevant hashtags

Example 2: Travel Image

• Uploading a scenic sea view image

• Receiving creative, engaging social media content such as:

  • Eye-catching Instagram reel titles

  • Emotional captions

  • YouTube video titles suitable for travel content

  • Optimized hashtags

These examples clearly show how the model adapts its output based on the visual context of each image.

In this lecture, you will get a clear and practical introduction to Google Cloud’s Document AI API, a powerful service designed to automatically understand, digitize, and extract information from documents using artificial intelligence.

This session focuses on concepts, architecture, and real-world use cases, helping you understand what Document AI is, how it works, and when to use it, before moving into hands-on implementations in later lectures.

What Is Document AI?

The lecture begins by explaining Document AI in simple, non-technical terms:

• Document AI is a Google Cloud service that uses AI to read, understand, and extract information from documents

• It works like a smart digital assistant that can process large volumes of documents automatically

• It converts unstructured documents (PDFs, scanned images, forms) into structured, usable data

You will see how Document AI eliminates the need for slow and error-prone manual data entry.

Why Document AI Is Important

You will learn why document processing is a critical challenge for businesses:

• Most business information is still stored in documents

• Manual document digitization is time-consuming and expensive

• Extracting information accurately at scale is difficult without AI

Common real-world examples discussed include:

• Invoice and receipt processing

• Medical intake forms

• Identity verification using ID cards

• Digitizing printed or handwritten documents

• Expense report validation

Document AI Architecture and Core Components

The lecture walks through the high-level architecture of Document AI, including:

• How Document AI sits on top of Vertex AI

• How it integrates with:

  • Google Cloud Storage

  • BigQuery

  • Vertex AI Search and Conversation

• How it supports scalable, end-to-end document processing pipelines

You will also understand how Document AI uses OCR, layout analysis, classification, and entity extraction together to process documents intelligently.

Key Document Processing Capabilities

This lecture explains the main capabilities supported by Document AI:

• Text and layout extraction (OCR)

• Key-value pair detection

• Table extraction

• Document classification

• Splitting and routing documents by type

• Entity extraction and normalization

• Multilingual document support

• Dataset preparation for training and evaluation

These capabilities allow businesses to automate complex document workflows without building custom ML models from scratch.

Understanding Document AI Processors

A major part of this lecture focuses on Document AI processors, which act as the bridge between:

• Raw document files

• Machine learning models that analyze them

You will learn:

• What a processor is

• Why processors are required

• Different processor categories such as:

  • Digitization

  • Extraction

  • Classification

• How to choose the right processor based on your use case

Examples include:

• Enterprise Document OCR for text extraction

• Form Parser for structured forms

• Invoice and receipt processors for financial documents

Typical Workflow with Document AI

The lecture outlines the standard steps to use Document AI:

1. Choose the appropriate processor

2. Create the processor in Google Cloud

3. Send documents for processing

4. Receive structured output such as:

   • Extracted fields

   • Tables

   • OCR text

   • Confidence scores

This gives you a clear mental model of how Document AI fits into real applications.

Supported File Types and Limitations

You will also learn about practical constraints, including:

• Supported document formats (PDF, PNG, JPEG, TIFF, GIF)

• Maximum file size limits

• Page limits per processor

• Language support differences across processors

Understanding these limits is essential before building production solutions.

Live Demonstration with Sample Documents

The lecture includes hands-on exploration using sample documents, such as:

• Invoices

• Personal information forms

You will see how Document AI:

• Identifies key-value pairs

• Extracts structured fields like dates, amounts, names, and addresses

• Extracts tables and OCR text

• Displays extracted information visually

This helps you clearly understand the quality and depth of extraction provided by Document AI.

In this lecture, you will move from understanding Document AI concepts to practically setting up Document AI in Google Cloud. This session focuses on the essential configuration steps required before using Document AI programmatically, ensuring you are fully prepared for hands-on implementation in upcoming tutorials.

What This Lecture Covers

In this tutorial, you will learn how to enable the Document AI API, create a document processor, and prepare Google Cloud Storage for document processing. These steps form the foundation of any real-world Document AI solution.

Step 1: Enabling the Document AI API

The lecture begins by guiding you through enabling the Cloud Document AI API from the Google Cloud Console. You will understand:

• Where to find the Document AI API in the Google Cloud API Library

• How to enable the API for your project

• What information is available on the API overview page, including:

  • Pricing details

  • Page processing limits

  • Quotas and system limits

  • API metrics and usage monitoring

• How Document AI integrates with other Google services such as:

  • Google Drive API

  • Gmail API

  • Gemini API

This step ensures your Google Cloud project is fully authorized to use Document AI services.

Step 2: Exploring Document AI in the Console

After enabling the API, you will explore the Document AI section in Google Cloud Console, where you will see:

• An overview of how Document AI transforms unstructured documents into structured data

• A guided workflow explaining:

  1. Creating a processor

  2. Extracting structured data

  3. Using the extracted data for downstream applications

This gives you a clear understanding of how Document AI works end to end.

Step 3: Understanding Document AI Processors

A major focus of this lecture is learning about Document AI processors, which are responsible for analyzing documents. You will explore:

Processor Gallery

• Prebuilt processors available for immediate use

• Categories such as:

  • General processors

  • Specialized processors

  • Trainable processors

Common Processor Types

• Document OCR – for extracting text and layout from documents

• Form Parser – for extracting key-value pairs, checkboxes, and structured fields

• Layout Parser – for identifying document structure and sections

Specialized Processors

You will also see processors designed for specific use cases, such as:

• Invoice processing

• Expense parsing

• Bank statement analysis

• Identity document verification

• Lending document classification

You will learn how to choose the right processor based on your business requirement.

Step 4: Creating a Document Processor

In this lecture, you will create a Form Parser processor step by step. You will understand:

• How to name and configure a processor

• How to choose the region for deployment

• Encryption options and why the default Google-managed encryption is sufficient in most cases

• Processor details such as:

  • Processor ID

  • Status

  • Processor type

  • Region

  • Prediction endpoint

This processor will later be used to extract structured data from uploaded documents.

Step 5: Testing the Processor with Sample Documents

Once the processor is created, you will test it directly from the console by uploading a sample PDF document. You will see:

• Automatic extraction of key-value pairs such as:

  • Name

  • Date of birth

  • Email

  • Phone number

  • Address

  • Signature

• Detection of entities like:

  • Person

  • Date

  • Email

  • Phone number

  • Address

• Exporting extracted results as structured JSON output

This live test demonstrates the power and accuracy of Document AI processors.

Step 6: Creating a Google Cloud Storage Bucket

To prepare for batch document processing, you will create a Google Cloud Storage (GCS) bucket. This section covers:

• Creating a new GCS bucket with proper region selection

• Understanding storage class options

• Using default access control and security settings

• Organizing data using folders

You will create:

• An input folder to store documents for processing

• An output folder to store processed results

Sample PDF documents are uploaded to the input folder, preparing the environment for automation.

In this lecture, you will begin the hands-on implementation of Document AI using a Google Colab notebook. This tutorial focuses on setting up the Colab environment, understanding the business problem, and preparing everything required before writing the core document processing logic.

This lecture acts as the bridge between Document AI setup in the Google Cloud Console and real-world programmatic usage.

Business Problem Overview

The lecture starts by clearly defining the business use case:

• You have PDF documents stored in a Google Cloud Storage bucket

• These documents need to be processed using a Document AI Form Parser processor

• The goal is to extract structured entities such as key-value pairs from the PDFs

• The final output should be a clean, structured Excel file containing the extracted information

This real-world scenario reflects common enterprise use cases such as form processing, data digitization, and automated document analysis.

Working in Google Colab

You will be introduced to the Google Colab notebook environment that will be used throughout the implementation. Colab is chosen because it:

• Runs in the cloud with no local setup required

• Integrates seamlessly with Google Cloud services

• Supports authentication with Google accounts

• Is ideal for experimentation and prototyping

Step 1: Environment Setup in Google Colab

The first major focus of this lecture is preparing the Colab environment so it can communicate with Google Cloud services.

You will understand why each dependency is required, including:

• A library to interact with Document AI

• A library to work with Google Cloud Storage

• A library for data manipulation and tabular processing

• A library to read and write Excel files

• Authentication libraries to securely access Google Cloud resources

The emphasis is on why these tools are needed rather than how to write the code.

Step 2: Authenticating Google Colab with Google Cloud

Next, the lecture explains how to authenticate Google Colab with your Google Cloud account.

You will learn:

• How Colab securely connects to your Google Cloud project

• Why authentication is mandatory for accessing Document AI and Cloud Storage

• What happens during the sign-in and permission approval process

• How this authentication allows the notebook to act on behalf of your project

This step ensures that your notebook has the required permissions to process documents.

Step 3: Project Configuration and API Enablement

After authentication, the lecture covers project-level configuration, including:

• Setting the correct Google Cloud project ID

• Verifying that the notebook is connected to the intended project

• Enabling required APIs such as:

  • Document AI API

  • Google Cloud APIs

  • Cloud Storage APIs

This ensures that all services used later in the workflow are available and properly configured.

Outcome of This Lecture

By the end of this tutorial, you will have:

• A clear understanding of the document processing business problem

• A fully configured Google Colab notebook

• All required libraries installed

• Secure authentication between Colab and Google Cloud

• The correct project and APIs enabled

This setup is essential before writing the logic to process PDFs, extract entities, and export results.

In this lecture, you will continue building your Document AI processing pipeline by completing Step 2: Configuration in the Google Colab environment. This tutorial focuses on setting up project-level configurations, authentication handling, storage paths, and client initialization, which are critical before processing any documents.

This lecture ensures that your notebook is fully connected, secure, and correctly aligned with your Google Cloud resources.

Recap of Previous Step

In the previous lecture, you successfully:

• Installed all required libraries in Google Colab

• Authenticated Colab with your Google Cloud account

• Enabled access to Document AI and Cloud Storage APIs

With authentication complete, this lecture moves into environment configuration.

Understanding the Configuration Phase

Configuration is the foundation that allows your notebook to:

• Locate the correct Google Cloud project

• Access the right Document AI processor

• Read PDF files from Cloud Storage

• Store outputs in structured locations

• Apply billing and quotas correctly

Without proper configuration, document processing workflows often fail or behave inconsistently.

Key Imports and Their Purpose

The lecture begins by explaining the required imports and why they matter:

• Operating system utilities to manage directories and file paths

• Data manipulation tools to prepare and analyze extracted information

• Typing utilities to improve readability and reduce errors in larger projects

This step emphasizes clarity, maintainability, and best practices.

Project Configuration Variables

Next, you configure all essential project-level details:

• Google Cloud Project ID to ensure all operations run in the correct project

• Region (location) aligned with the Document AI processor and storage bucket

• Processor ID that uniquely identifies the Form Parser processor you created earlier

This ensures your requests are routed to the correct Document AI resources.

Google Cloud Storage Paths

You will define and understand three important Cloud Storage locations:

• Input path – where PDF documents are stored

• Output path – where processed JSON results will be written

• Excel output path – where the final structured Excel files will be saved

These paths help keep the pipeline organized and scalable.

Local Working Directory in Colab

The lecture also covers setting up a local working directory inside the Colab environment:

• Used for temporary downloads and file inspection

• Helpful for debugging and verification

• Created dynamically to avoid manual setup

This allows you to work with files locally while still using cloud storage.

Authentication and Credential Management

A major focus of this lecture is secure credential handling, including:

• Loading default Google Cloud credentials

• Refreshing expired credentials automatically

• Applying correct permission scopes

• Setting the quota project to avoid billing conflicts

This step is especially important in multi-project or enterprise environments.

Client Initialization

You will then initialize the core service clients:

• Cloud Storage client to upload and download files

• Document AI client to send documents for processing

These clients establish the connection between your Colab notebook and Google Cloud services.

Processor Resource Path

The lecture explains how a fully qualified processor resource path is constructed:

• Combines project ID, region, and processor ID

• Acts as the unique identifier for Document AI requests

• Ensures documents are processed by the correct processor

This step confirms that your environment is ready for real processing tasks.

In this lecture, you will complete Step 3 of the Document AI pipeline by building helper utilities for Google Cloud Storage (GCS). These helper functions are designed to simplify and standardize common storage operations that are repeatedly used in document processing workflows.

By the end of this lecture, your project will have a clean and reusable way to interact with Cloud Storage, making the entire pipeline more readable, reliable, and scalable.

Recap of Progress So Far

Before this lecture, you have successfully completed:

• Step 1: Installing required libraries and authenticating Google Colab with Google Cloud

• Step 2: Configuring the project, processor, credentials, and storage paths

With authentication and configuration in place, the next logical step is to simplify Cloud Storage operations.

Why GCS Helper Functions Are Important

When working with Document AI, you often need to:

• Read PDF files from Cloud Storage

• Upload local files to a bucket

• List multiple documents for batch processing

• Parse Cloud Storage paths consistently

Writing this logic repeatedly can make notebooks hard to maintain. Helper utilities solve this problem by encapsulating repetitive tasks into reusable functions.

Goals of the GCS Helper Utilities

The helper utilities created in this lecture focus on four main objectives:

• Parsing Google Cloud Storage URIs

• Listing files inside a bucket using a path prefix

• Uploading files from Colab or a local environment to GCS

• Simplifying repetitive file-handling tasks

These utilities act as the backbone for all upcoming document processing steps.

Helper Function 1: Parsing GCS URIs

The first utility focuses on parsing Google Cloud Storage addresses.

This helper:

• Takes a GCS URI as input

• Separates the bucket name from the file or folder path

• Makes it easier to work with storage APIs that require these components separately

This step ensures that GCS paths are handled consistently across the notebook.

Helper Function 2: Listing Files in a GCS Bucket

The second utility helps you discover documents stored in Cloud Storage.

This helper:

• Searches for files inside a bucket

• Filters files using a specific folder or prefix

• Returns a list of matching files for further processing

This is especially useful for batch processing, where multiple PDF documents need to be sent to Document AI at once.

Helper Function 3: Uploading Files to Cloud Storage

The third utility simplifies uploading local files to Google Cloud Storage.

This helper:

• Takes a file from your local machine or Colab environment

• Uploads it to a specified bucket and folder

• Ensures files are available for Document AI processing

This step is helpful when testing new documents or dynamically adding files to the pipeline.

Benefits of Using GCS Helper Utilities

By introducing these helper functions, you gain:

• Cleaner and more readable notebooks

• Reduced code duplication

• Fewer errors related to file paths

• Easier debugging and maintenance

• A scalable structure for production pipelines

These utilities allow you to focus on document processing logic instead of low-level storage handling.

Completion of Step 3

At the end of this lecture, you have successfully:

• Created reusable utilities for Cloud Storage

• Standardized how GCS paths are parsed and managed

• Simplified file listing and upload operations

This completes Step 3: GCS Helpers in the Document AI workflow.

In this lecture, you will learn how to run the Document AI Form Parser in batch mode on multiple PDF files stored in Google Cloud Storage (GCS). This is a critical step when working with real-world document processing systems, where handling documents one by one is inefficient and costly.

By the end of this lecture, you will understand how to process multiple documents simultaneously, extract structured data, and store the results in a scalable and organized way.

What Problem This Lecture Solves

In real business scenarios, documents such as:

• Personal information forms

• Job application forms

• Invoices, receipts, or contracts

are usually stored in cloud storage and arrive in large volumes. Processing them individually is slow and expensive. This lecture demonstrates how to solve that problem using batch document processing with Document AI.

Objective of This Lecture

In this tutorial, you will:

• Process multiple PDF files at once from a GCS bucket

• Use the Form Parser processor created earlier

• Automatically extract structured entities from each document

• Store the extracted results as JSON files in Cloud Storage

Input Documents Used

The lecture works with two sample PDF files that were already uploaded in earlier steps:

• Personal Information Form

• Job Application Form

These documents are stored inside the input folder of a GCS bucket and are ready for batch processing.

Why Batch Processing Is Important

This step is performed to achieve the following goals:

1. Discover All Ready-to-Process PDFs

The system automatically finds all PDF documents stored in the input folder, eliminating the need to manually specify each file.

2. Clean Previous Results

Before starting a new batch job, old output files are removed. This prevents confusion and ensures that only fresh and accurate results are generated.

3. Process Multiple Documents Together

Instead of sending one document at a time, all PDFs are sent together to Document AI for simultaneous processing, which improves speed and efficiency.

4. Handle Large-Scale Operations

Batch processing is:

• Faster than individual processing

• More cost-effective

• Suitable for enterprise-scale document workflows

What Happens During Batch Processing

During this step:

• All PDF files are collected from the GCS input folder

• Old output data is cleaned automatically

• Documents are submitted to Document AI in a single batch job

• Document AI processes each PDF using the Form Parser

• Extracted entities are saved as JSON files, one per document

Understanding the Output

After batch processing completes:

• A new output folder is created in the GCS bucket

• Separate subfolders are generated for each document

• Each subfolder contains a JSON file with extracted data

These JSON files include:

• Key-value pairs such as names, dates, email addresses, phone numbers

• Structured fields like application ID, address details, signatures, and more

• OCR-based text extraction and entity detection

Verifying the Results

In this lecture, you also learn how to:

• Open and inspect the generated JSON files

• Match extracted entities with the original PDF content

• Verify that fields such as names, dates, email IDs, phone numbers, and addresses are correctly extracted

This validation step confirms that the Form Parser is working accurately.

In this lecture, we focus on Step 5 of the Document AI workflow: creating parsing helper functions. This step is essential for converting the raw output generated by Document AI into a format that is easy to understand, analyze, and store.

After batch processing, Document AI produces complex JSON files with deeply nested structures. These files are powerful but not directly usable for business analytics or reporting. In this tutorial, you will learn how to design helper functions that extract, flatten, and organize this data into clean, structured formats.

Why This Step Is Important

Document AI output is:

• Highly structured

• Deeply nested

• Designed for machine processing, not direct human use

To make this data usable in:

• Excel reports

• Databases

• Dashboards

• Downstream machine learning pipelines

we must first parse and simplify it. This lecture shows you how to do exactly that.

Objective of This Lecture

By the end of this tutorial, you will understand how to:

• Read Document AI output files from Google Cloud Storage

• Convert layout-based text into readable content

• Flatten complex entity structures into simple rows

• Extract key-value pairs in a clean, standardized format

These parsing helpers form the backbone of any production-grade document processing pipeline.

Overview of Helper Functions Created

In this lecture, we define four essential helper functions, each with a clear and focused responsibility.

1. Converting Layout Objects to Readable Text

Document AI stores extracted text using layout references and index positions.
The first helper function:

• Converts layout objects back into readable text

• Uses index ranges to reconstruct meaningful text segments

• Makes extracted content human-readable

This step is critical because most entities refer to text indirectly through layout indexes.

2. Loading Document AI Output from GCS

After batch processing, results are saved as JSON files in a GCS output bucket.

The second helper function:

• Downloads the JSON output files from Google Cloud Storage

• Loads them into Python-friendly data structures

• Makes the extracted results accessible for further processing

This bridges the gap between cloud-based processing and local analysis.

3. Flattening Nested Entities

Document AI entities often contain:

• Parent-child relationships

• Nested sub-entities

• Repeated structures

The third helper function:

• Recursively traverses these nested entities

• Flattens them into a simple list of rows

• Prepares the data for spreadsheets or databases

This transformation is crucial for exporting data into tabular formats like Excel.

4. Extracting Key-Value Pairs

The final helper function serves as the main entry point for structured data extraction.

It:

• Collects all detected entities

• Applies flattening logic

• Outputs clean, structured rows containing key-value information

This makes it easy to extract meaningful business data such as:

• Names

• Dates

• IDs

• Addresses

• Contact details

In this lecture, we complete the final step of our Document AI mini-project by converting the processed JSON output into clean, easy-to-use Excel files.

In the previous tutorial, we created helper functions to parse and organize the Document AI output. Now, we use those helpers to generate one Excel file per document, containing only the extracted entities and their confidence scores.

This step transforms raw AI output into a business-friendly format that can be shared with analysts, operations teams, or stakeholders.

Objective of This Lecture

By the end of this tutorial, you will understand how to:

• Locate Document AI output JSON files stored in Google Cloud Storage

• Process each JSON file independently

• Extract only meaningful entities from each document

• Create one Excel file per document

• Upload the generated Excel files back to Google Cloud Storage

This completes an end-to-end document processing pipeline, from PDFs to structured Excel reports.

Understanding the Input and Output

Input

• PDF documents stored in a GCS input folder

• Document AI output stored as JSON files in a GCS output folder

• Each JSON file corresponds to one PDF document

Output

• One Excel file per document

• Each Excel file contains:

  • Field Name

  • Field Value

  • Confidence Score

This makes the output easy to review, validate, and analyze.

Step-by-Step Workflow Covered in This Lecture

1. Identifying JSON Output Files

• The process begins by locating all JSON files created during batch processing

• These files are automatically discovered from the GCS output folder

• The system confirms how many documents were processed

This ensures no document is missed.

2. Initializing Per-Document Processing

• A tracking mechanism is set up to monitor generated Excel files

• Each JSON file is processed independently

• This guarantees one Excel file per input PDF

3. Extracting Entities from JSON Files

For each JSON file:

• The Document AI output is loaded from GCS

• Parsed entity data is extracted using previously created helper functions

• Only key-value entities are selected

Non-essential metadata is ignored to keep the Excel file clean.

4. Generating Meaningful Excel File Names

• Excel files are named based on the original PDF file

• This makes it easy to map:

  • PDF → JSON → Excel

Clear naming is important for traceability in production pipelines.

5. Creating Excel Files

• Each document gets a single-sheet Excel file

• The sheet contains:

  • Entity name

  • Extracted value

  • Confidence score

This format is ideal for validation and business reporting.

6. Uploading Excel Files to GCS

• All generated Excel files are uploaded back to Google Cloud Storage

• A dedicated folder is created for Excel outputs

• Files are now accessible from anywhere in the cloud

This enables easy sharing and integration with other systems.

Final Results Demonstrated

At the end of this lecture, you will see:

• PDF files in the input folder

• JSON files containing raw Document AI output

• Excel files containing clean, structured entity data

• One Excel file per document

This confirms a successful end-to-end processing flow.

In this lecture, we rebuild our Document AI Form Parser project, but this time with an interactive user interface using Gradio. Instead of processing documents only through scripts, we create a user-friendly web app where users can upload a PDF, extract entities, and download the results as an Excel file.

This approach demonstrates how AI-powered document processing can be exposed to end users through a simple and intuitive interface.

What This Project Does

In this application:

• A user uploads a single PDF form using a web interface

• Google Document AI Form Parser extracts entities from the document

• Extracted entities are organized into structured fields

• Results are exported into an Excel file

• The Excel file can be downloaded directly from the UI

This mimics a real-world document automation tool used in HR, finance, compliance, and operations teams.

High-Level Architecture

The project integrates three key components:

• Google Document AI – for intelligent entity extraction

• Pandas & Excel export – for structured output

• Gradio – for building a simple web-based user interface

Together, they create a complete end-to-end document processing application.

Step-by-Step Breakdown of the Workflow

Step 1: Installing Required Libraries

We begin by installing all necessary libraries for:

• Document AI communication

• Authentication

• Data processing and Excel generation

• UI creation using Gradio

This ensures the environment is fully prepared.

Step 2: Imports and Basic Setup

Core Python utilities are imported for:

• File handling

• Debugging and error tracking

• Data manipulation

• Type safety and readability

This makes the project easier to maintain and debug.

Step 3: Authentication in Google Colab

Since the application runs inside Google Colab, user authentication is required:

• The user logs in with a Google account

• Secure access is granted to Google Cloud services

• The notebook can now call the Document AI API

Step 4: Google Cloud & Document AI Configuration

In this step:

• The Google Cloud project is configured

• The processor location and processor ID are defined

• The Document AI client is initialized

• Authentication credentials are refreshed if required

This connects the application directly to the Form Parser processor.

Step 5: Entity Extraction Helper Functions

Document AI returns structured but complex data.
To simplify this:

• Helper functions flatten nested entities

• Output is standardized into three columns:

  • Field Name

  • Field Value

  • Confidence

This format is ideal for Excel and reporting.

Step 6: Reading Uploaded PDF Files

A helper function is used to:

• Accept uploaded PDFs from different sources

• Convert them into raw bytes

• Ensure flexibility and reliability for file uploads

This allows smooth integration with the Gradio interface.

Step 7: Core Processing Pipeline

This is the heart of the application. The pipeline performs:

• PDF validation

• Entity extraction using Document AI

• Conversion of results into a structured table

• Exporting the output as an Excel file

• Returning status messages for success or failure

Robust error handling ensures clear feedback to the user.

Step 8: Building the Gradio User Interface

A simple but powerful UI is created where users can:

• Upload a PDF form

• Click a button to extract entities

• Preview extracted data in a table

• Download the Excel file

• View success or error messages

This removes the need for technical knowledge from the end user.

Step 9: Launching the Application in Colab

The Gradio app is launched with:

• A public shareable URL

• A Colab-compatible fallback option

This ensures the app works reliably inside notebooks and browsers.

In this lecture, we begin our journey into AutoML for tabular data using Google Cloud and Vertex AI. This tutorial focuses on preparing data correctly, which is one of the most critical steps before training any AutoML model.

You will learn how to move from a raw CSV file to well-structured training, testing, and evaluation datasets, ready to be used in Vertex AI AutoML.

What You Will Learn in This Lecture

In this tutorial, you will learn how to:

• Prepare tabular data for AutoML workflows

• Use Google Cloud Storage to manage datasets

• Create datasets and tables in BigQuery from CSV files

• Understand a real-world binary classification problem

• Split data into training, testing, and evaluation sets in a stable and reproducible way

Business Problem Overview

We work with an Employee Attrition dataset, where the goal is to predict:

• Whether an employee will stay with the company or leave

Key points:

• The target column is attrition

• This is a binary classification problem

• All other columns act as feature variables

This is a very common business use case in HR analytics and workforce planning.

Step-by-Step Workflow Covered in This Lecture

1. Creating a Google Cloud Storage (GCS) Bucket

We start by creating a dedicated GCS bucket:

• The bucket is created in the US Central 1 region

• A folder named input is created inside the bucket

• The employee attrition CSV file is uploaded to this folder

This bucket will serve as the data source for BigQuery and Vertex AI.

2. Creating a BigQuery Dataset

Next, we move to BigQuery and:

• Create a new dataset for AutoML

• Ensure the dataset is created in the same region as the GCS bucket

• This region consistency is important for smooth integration with Vertex AI

3. Creating a BigQuery Table from the CSV File

Using the uploaded CSV file:

• A BigQuery table is created directly from Google Cloud Storage

• Schema detection is handled automatically

• No partitioning is applied for simplicity

Once created, the table is explored to:

• Review column names and data types

• Preview sample records

• Identify the target column (attrition)

4. Understanding the Dataset Structure

At this stage:

• The attrition column is identified as the label

• All remaining columns are treated as features

• The problem is confirmed as binary classification

This clarity is essential before moving into AutoML training.

5. Creating Stable Data Splits (Train, Test, Evaluation)

Instead of random splits, a deterministic and reproducible splitting strategy is used:

a. Split Key Table

• A separate table is created to generate a stable split key for each employee

• The split key is a numeric value between 0 and 1

• This ensures consistent splits every time the pipeline is run

b. Training Table

• Approximately 70% of the data

• Used to train the AutoML model

c. Testing Table

• Approximately 15% of the data

• Used to evaluate model performance during training

d. Evaluation Table

• Remaining 15% of the data

• Used for final unbiased evaluation

This three-way split follows best practices for machine learning.

6. Validating the Data Splits

Finally, the lecture verifies:

• Row counts in each dataset

• Percentage distribution across train, test, and evaluation tables

This confirms that:

• The data is split correctly

• There is no overlap

• The distribution closely matches the intended ratios

In this lecture, we move from data preparation to model training using Vertex AI AutoML for tabular data. After successfully creating the training, testing, and evaluation tables in the previous tutorial, we now focus on building a fully managed machine learning model without writing any training code.

This session demonstrates how to create a Vertex AI Dataset, connect it to BigQuery, and train an AutoML classification model step by step using the Google Cloud Console.

Step-by-Step Workflow Covered in This Lecture

1. Navigating to Vertex AI

We begin by opening Vertex AI in the Google Cloud Console and selecting the correct region (US Central 1) to match our dataset and BigQuery tables.

2. Creating a Vertex AI Dataset

We create a new dataset with the following configuration:

• Dataset Type: Tabular

• Objective: Regression and Classification

• Encryption: Google-managed encryption key

This dataset will serve as the input source for AutoML training.

3. Connecting BigQuery as the Data Source

Instead of uploading CSV files again, we directly connect:

• The BigQuery training table (employee_train)

• Dataset format is automatically recognized as BigQuery

• Dataset metadata such as location, encryption, and data types is displayed

This approach is scalable and commonly used in production environments.

4. Generating Dataset Statistics

Once the dataset is created:

• Vertex AI automatically computes column statistics

• You can see:

  • Number of distinct values per column

  • Data type distribution (boolean, integer, string, etc.)

• This helps validate data quality before training

You can also view dataset lineage using graphical or list views.

5. Starting AutoML Model Training

Next, we start training a new model:

• Training Method: AutoML

• Model Type: Tabular Classification

• Target Column: attrition

• Model Name: AutoML Tabular Model

This tells Vertex AI that we want to predict employee attrition using AutoML.

6. Understanding Data Splitting During Training

Although we already created train, test, and evaluation tables earlier:

• AutoML still performs an internal split on the training dataset

• Default split configuration:

  • 80% Training

  • 10% Validation

  • 10% Testing

• The split method used is random

Other options such as manual or chronological splits are available, but default settings are sufficient for most use cases.

7. Training Configuration and Optimization

During training setup:

• Feature transformations are handled automatically

• Optimization objective defaults to ROC AUC

• Feature Store integration is optional and skipped

• Encryption remains Google-managed

All settings follow AutoML best practices, allowing the platform to automatically tune the model.

8. Setting the Training Budget

Since our dataset contains fewer than 10,000 rows:

• Training budget is set between 1 to 3 hours

• We select 1 hour

• Early stopping is enabled by default

Vertex AI provides a clear pricing estimate before training begins, ensuring transparency in cost.

9. Monitoring Model Training

Once training starts:

• The job appears under Vertex AI → Training

• You can monitor:

  • Training status

  • Pipeline ID

  • Model type (Tabular Classification)

  • Budget and elapsed time

  • Region and timestamps

Training typically takes up to a few hours, depending on dataset size and complexity.

In this lecture, we complete the AutoML Tabular machine learning lifecycle by exploring the trained model, analyzing its performance, and deploying it to a Vertex AI endpoint for real-time predictions.

After the AutoML training finishes successfully, this tutorial focuses on model evaluation, interpretability, deployment, and testing, which are critical steps before using any machine learning model in production.

What You Will Learn in This Lecture

By the end of this session, you will be able to:

• Locate your trained AutoML model in the Vertex AI Model Registry

• Understand key evaluation metrics for a classification model

• Interpret confusion matrix, ROC curve, and precision-recall curve

• Analyze feature importance generated by AutoML

• Deploy the model to a Vertex AI endpoint

• Test the deployed endpoint with real input values

• Interpret prediction results and confidence scores

Step-by-Step Workflow Covered in This Lecture

1. Accessing the Trained Model

Once training is completed:

• The model moves from Training Jobs to the Model Registry

• From here, you can manage, deploy, and monitor the model

• Clicking the model opens detailed evaluation and deployment options

2. Understanding the Evaluation Metrics

Inside the Evaluation tab, Vertex AI provides multiple performance metrics:

• Confidence Threshold (0.5) – used to convert probabilities into predictions

• Accuracy

• Precision

• Recall

• F1 Score

• ROC AUC

• Log Loss

These metrics help you understand how well the model predicts employee attrition.

3. Visual Performance Analysis

Vertex AI also provides rich visualizations, including:

• ROC Curve – shows the trade-off between true positive and false positive rates

• Precision-Recall Curve – useful when dealing with class imbalance

• Threshold Curves – help adjust prediction confidence levels

These graphs help you choose the right confidence threshold for business use cases.

4. Confusion Matrix Interpretation

The confusion matrix clearly shows:

• True Positives

• True Negatives

• False Positives

• False Negatives

Using this matrix, we can manually calculate the model accuracy, which comes out to approximately 57–60%. This confirms the evaluation score shown by Vertex AI.

Note: The accuracy is relatively low because the dataset is small. With larger and more balanced data, AutoML generally performs much better.

5. Feature Importance Analysis

Vertex AI automatically calculates feature importance, helping you understand:

• Which features most influence predictions

• How strongly each feature impacts the model

In this case:

• Features like OverTime, JobLevel, and other employment-related attributes rank highest

• This improves model explainability and trust

6. Deploying the Model to an Endpoint

Next, we deploy the model for online predictions:

• Create a new endpoint

• Select the region (US Central 1)

• Assign 100% traffic to this model

• Choose compute resources (machine type, CPU, memory)

• Skip model monitoring for now

Once deployed, the endpoint becomes active and ready to receive prediction requests.

7. Exploring the Endpoint Dashboard

After deployment, you can monitor:

• Request count

• Latency

• Error rate

• Response time

• Model inference metrics

Initially, these metrics show no data because the endpoint is newly created.

8. Testing the Deployed Model

To test the endpoint:

• Provide input values for all required features

• Submit a prediction request

• Receive:

  • Predicted label (true or false)

  • Confidence score

In this case:

• Prediction = False

• Meaning: the employee is not likely to leave the organization

This confirms that the endpoint is working correctly.

In this lecture, we move beyond online predictions and learn how to perform batch inference using a trained AutoML Tabular model in Vertex AI. Batch inference is a critical capability when you need to generate predictions for large datasets at once, instead of making real-time requests through an endpoint.

You will learn how to configure, run, and analyze batch prediction jobs using BigQuery tables as input and Google Cloud Storage (GCS) as output.

What You Will Learn in This Lecture

By the end of this lecture, you will be able to:

• Understand the difference between online inference and batch inference

• Create a batch inference job using a BigQuery table

• Configure batch prediction output in CSV format

• Store prediction results in a GCS bucket

• Explore and interpret batch prediction output files

• Understand available input and output options for batch inference in Vertex AI

Step-by-Step Topics Covered

1. Why Batch Inference Is Important

Batch inference is used when:

• You need predictions for thousands or millions of records

• Real-time latency is not required

• Predictions are needed for reporting, analytics, or downstream processing

• Cost efficiency is more important than low-latency responses

In this tutorial, we generate predictions for the employee test dataset created earlier.

2. Creating a Batch Inference Job

You begin by creating a new batch inference job and configuring:

• Batch inference name for easy tracking

• Input source as a BigQuery table (employee_test)

• Output format as CSV

• Output location as a Google Cloud Storage bucket

This setup allows Vertex AI to process all test records automatically and store predictions in cloud storage.

3. Input and Output Configuration

Key configuration choices explained in this lecture include:

• Input source options

  • BigQuery tables

  • CSV files from Cloud Storage

• Output format options

  • CSV files

  • BigQuery tables

  • JSON format

  • TFRecord format

For this demo, we use:

• BigQuery as the input

• CSV files in GCS as the output

4. Running the Batch Inference Job

Once configured:

• Vertex AI creates a batch inference pipeline

• The job runs asynchronously

• The status changes from Running to Completed

• Multiple output files are generated automatically

This approach allows efficient and scalable predictions without manual intervention.

5. Exploring Batch Prediction Results

After the job completes:

• The output folder in GCS contains multiple CSV files

• Files are split automatically for scalability

• Each CSV contains:

  • Prediction probabilities for each class

  • Separate scores for:

    • Attrition_False_Score

    • Attrition_True_Score

This makes it easy to:

• Analyze prediction confidence

• Choose custom decision thresholds

• Integrate results into analytics or dashboards

6. Understanding Prediction Output

Each record includes:

• Probability that the employee will stay

• Probability that the employee will leave

These scores help business teams:

• Identify high-risk employees

• Take preventive actions

• Perform trend analysis at scale

7. Where Batch Inference Fits in the ML Lifecycle

This lecture completes an important phase of the ML workflow:

• Model training

• Model evaluation

• Online inference (real-time predictions)

• Batch inference (large-scale predictions)

Batch inference is commonly used in:

• HR analytics

• Marketing segmentation

• Financial risk assessment

• Periodic reporting pipelines

In this lecture, we focus on an important but often ignored step in machine learning projects—resource cleanup. After completing the AutoML Tabular workflow in Vertex AI, it is critical to delete unused resources to avoid unnecessary cloud costs, keep the project organized, and follow cloud best practices.

You will learn how to safely delete all resources created during the AutoML Tabular project, including Vertex AI components, BigQuery datasets, and Cloud Storage files.

Why Resource Cleanup Is Important

Machine learning services on Google Cloud are billable resources. If left running, they can continue to incur costs even when not in use. This lecture helps you:

• Prevent unexpected cloud charges

• Maintain a clean and manageable Google Cloud project

• Follow production-grade cloud hygiene practices

• Understand dependencies between cloud resources

What You Will Learn in This Lecture

By the end of this lecture, you will be able to:

• Correctly delete a deployed Vertex AI endpoint

• Remove trained AutoML Tabular models and training pipelines

• Delete Vertex AI datasets

• Clean up BigQuery datasets and tables

• Remove input and output files from Google Cloud Storage

• Understand the correct order of deletion to avoid errors

Step-by-Step Topics Covered

1. Cleaning Up Vertex AI Resources

You start by cleaning all resources created in Vertex AI, following the correct order:

a. Deleting the Endpoint

• Endpoints must be deleted before deleting models or pipelines

• A model must be undeployed from the endpoint first

• Once undeployed, the endpoint can be safely deleted

b. Deleting the Training Pipeline

• After the endpoint is removed, the training pipeline can be deleted

• This removes the AutoML training job and related metadata

c. Deleting the Vertex AI Dataset

• The dataset created for AutoML training is deleted next

• This ensures no unused datasets remain in the project

2. Deleting BigQuery Resources

Next, you clean up BigQuery:

• Delete the dataset created for AutoML tabular data

• This automatically removes:

  • Training tables

  • Test tables

  • Evaluation tables

This step ensures no unused BigQuery storage remains active.

3. Cleaning Up Cloud Storage (GCS)

Finally, you remove Cloud Storage resources:

• Delete the input folder containing uploaded CSV files

• Delete the output folder containing batch prediction results

• Ensure the bucket is fully cleaned

This completes the removal of all storage resources created during the project.

Best Practices Highlighted in This Lecture

• Always undeploy models before deleting endpoints

• Follow the correct deletion order to avoid permission errors

• Clean up resources immediately after experimentation

• Regularly review your cloud environment for unused assets

In this lecture, we begin our journey into AutoML Forecasting using Google Vertex AI. Forecasting is a powerful machine learning technique used to predict future values based on historical time-series data, such as stock prices, sales, demand, or traffic trends.

In this session, you will learn how to prepare data, create datasets, and configure an AutoML Forecasting model step by step using Google Cloud services.

What You Will Learn in This Lecture

By the end of this lecture, you will be able to:

• Prepare time-series data for forecasting

• Upload forecasting data to Google Cloud Storage (GCS)

• Create BigQuery datasets and tables for AutoML forecasting

• Understand required columns for forecasting models

• Create a Vertex AI Tabular Forecasting dataset

• Configure and start training an AutoML Forecasting model

Key Topics Covered

1. Uploading Forecasting Data to Cloud Storage

• Use an existing GCS bucket created earlier

• Create an input folder for organizing forecasting files

• Upload:

  • Historical data file (training data)

  • Test data file (future timestamps with missing target values)

This ensures clean separation between training and prediction data.

2. Creating a BigQuery Dataset for Forecasting

• Create a new dataset dedicated to forecasting

• Select the same region (us-central1) to maintain compatibility

• Organize forecasting tables inside a single dataset

This dataset will act as the data source for Vertex AI.

3. Creating BigQuery Tables from CSV Files

You will create multiple tables:

• Historical stock data table

  • Contains columns like date, open, high, low, close, and volume

• Test data table

  • Contains future dates where the target value needs to be predicted

• Training table

  • A cleaned and simplified version used specifically for AutoML forecasting

4. Understanding Required Columns for Forecasting

For AutoML Forecasting, you must define:

• Timestamp column
  Represents the time dimension (e.g., daily dates)

• Target column
  The value you want to forecast (e.g., closing stock price)

• Series identifier column
  Identifies each time series

  • Even if there is only one series, a placeholder ID is required

This lecture explains why these columns are mandatory and how they are used internally by AutoML.

5. Creating a Vertex AI Forecasting Dataset

• Navigate to Vertex AI → Datasets

• Choose Tabular → Forecasting

• Select BigQuery table as the data source

• Specify:

  • Series identifier column

  • Timestamp column

Once created, the dataset becomes ready for model training.

6. Configuring AutoML Forecasting Model Training

You will learn how to configure key forecasting parameters, including:

• Target column (value to predict)

• Time granularity (daily data)

• Forecast horizon
  Number of future time periods to predict

• Context window
  Amount of historical data used for predictions

• Holiday region
  Helps the model learn seasonal effects

• Chronological data split
  Ensures proper time-based training, validation, and testing

7. Training Budget and Model Execution

• Select AutoML as the training method

• Allocate training budget based on dataset size

• Start the AutoML Forecasting training job

• Monitor the training status in Vertex AI

The training process runs asynchronously and may take up to a few hours depending on data size and configuration.

In this lecture, we continue our AutoML Forecasting journey by exploring the trained forecasting model, understanding its evaluation metrics, and generating predictions using batch inference in Vertex AI.

You will learn why forecasting models work differently from other machine learning models and how to generate predictions correctly using Google’s managed forecasting workflow.

What You Will Learn in This Lecture

By the end of this lecture, you will be able to:

• Explore a trained AutoML Forecasting model in Vertex AI

• Understand key forecasting evaluation metrics

• Analyze feature importance for time-series models

• Learn why forecasting models cannot be deployed to online endpoints

• Generate predictions using batch inference

• Access and interpret forecasting results stored in BigQuery

Key Topics Covered

1. Reviewing the Trained AutoML Forecasting Model

• The forecasting model is successfully trained using Vertex AI

• Training duration is close to two hours, based on the allocated budget

• The model appears in the Model Registry under training pipelines

• Model type: Tabular Forecasting

This confirms that the AutoML forecasting workflow has completed successfully.

2. Understanding Model Evaluation Metrics

Inside the Evaluate tab, you analyze forecasting performance using industry-standard metrics:

• MAE (Mean Absolute Error) – Measures average absolute error

• MAPE (Mean Absolute Percentage Error) – Measures percentage-based error

• RMSE (Root Mean Squared Error) – Penalizes large errors

• RMSLE (Root Mean Squared Log Error) – Useful for scale-sensitive data

• R² Score – Indicates how well the model explains variance

These metrics help you judge how reliable the forecast is.

3. Feature Importance in Forecasting Models

• Feature importance shows how much each feature contributes to predictions

• Date (time) contributes the most

• Target value (close price) also plays a significant role

This confirms that the model is learning meaningful time-based patterns.

4. Why Forecasting Models Cannot Be Deployed to Endpoints

• Unlike classification or regression models, AutoML Forecasting models cannot be deployed for real-time predictions

• Attempting to deploy results in an error

• Forecasting predictions must be generated using batch inference only

This is an important architectural difference to understand.

5. Automatically Exported Evaluation Predictions

• During training, the option to export test data to BigQuery was enabled

• Vertex AI automatically creates a BigQuery table containing:

  • Actual values

  • Predicted values

  • Prediction timestamps

  • Series identifiers

This allows easy inspection and validation of model predictions.

6. Creating a Batch Inference Job

You then create a batch prediction job by:

• Selecting the trained forecasting model

• Choosing a BigQuery test table as input

• Specifying BigQuery as the output destination

• Disabling optional explainability and monitoring

• Submitting the batch inference job

Batch inference is the correct and recommended way to generate forecasts.

7. Monitoring Batch Prediction Execution

• The batch inference job runs for around 20 minutes

• Job status shows:

  • Number of predictions processed

  • Success and failure counts

  • Execution time

• Logs are available for auditing and debugging

8. Viewing Forecasting Results in BigQuery

After completion:

• A new BigQuery table is created automatically

• This table includes:

  • Date

  • Predicted forecast value

  • Series identifier

• Each row represents a future forecast point

These predictions can now be used for reporting, dashboards, or downstream analysis.

In this lecture, we begin learning how to work with text data on Google Cloud, focusing on sentiment analysis using Vertex AI. You will understand why traditional AutoML Text models are no longer supported and how Gemini models are now used instead for text classification tasks.

This lecture walks you through the complete transition from AutoML Text to Gemini-based fine-tuning, using a real-world example of restaurant reviews.

What You Will Learn in This Lecture

By the end of this lecture, you will be able to:

• Understand why AutoML Text is deprecated in Vertex AI

• Learn how text classification is now handled using Gemini models

• Prepare text datasets for supervised fine-tuning

• Understand the required JSON format for Gemini fine-tuning

• Fine-tune a Gemini model for sentiment classification

• Use trained models for batch predictions

Key Topics Covered

1. Understanding the Text Classification Use Case

• The dataset contains restaurant reviews

• Each review has a label:

  • 1 → Positive review

  • 0 → Negative review

• The dataset includes 1,000 total records

• This represents a binary text classification problem

2. Uploading Text Data to Google Cloud Storage

• A TSV (tab-separated) file is uploaded to a GCS bucket

• The file contains:

  • Review text

  • Sentiment label

• This data will be used for training and prediction

3. AutoML Text Deprecation Explained

• Attempting to create a Text dataset in Vertex AI results in an error

• Google has deprecated AutoML NLP models

• Vertex AI now recommends using:

  • Gemini models

  • Vertex AI Studio

  • Generative AI workflows

This is an important architectural change you must understand when working with text data on Google Cloud.

4. Using Gemini for Text Classification

• Instead of AutoML Text, you use Gemini foundation models

• Gemini can classify text using prompt-based instructions

• You define:

  • What is a positive review

  • What is a negative review

• The model responds with:

  • 1 for positive sentiment

  • 0 for negative sentiment

5. Why Fine-Tuning Is Required

• Prompt-based classification works, but fine-tuning improves:

  • Accuracy

  • Consistency

  • Efficiency

• Fine-tuning is ideal when:

  • You have labeled data

  • The task is clearly defined

• Text classification is a perfect use case for supervised fine-tuning

6. Preparing Data for Gemini Fine-Tuning

• The original TSV file is converted into three JSON files:

  • Training data

  • Validation data

  • Batch prediction data

• Each JSON record includes:

  • System instruction (defines model behavior)

  • User content (review text)

  • Model response (label: 0 or 1)

This format teaches Gemini exactly how to classify sentiment.

7. Understanding the Gemini Fine-Tuning JSON Structure

Each record clearly defines:

• System role → Explains the classification rules

• User role → Contains the review text

• Model role → Contains the correct label

This structure is required for supervised tuning.

8. Fine-Tuning a Gemini Model in Vertex AI

• A fine-tunable Gemini model is selected from Model Garden

• A new tuned model is created with:

  • A base Gemini model

  • Training data from Cloud Storage

  • Validation data from Cloud Storage

• Default training parameters are used

• Fine-tuning runs for more than one hour

9. Monitoring the Fine-Tuning Process

• The model status shows as Running

• Vertex AI manages the entire training pipeline

• Once completed, the tuned model will be available for predictions

In this lecture, we continue working with the fine-tuned Gemini text classification model that was trained in the previous tutorial. Now that the tuning process has completed successfully, we focus on understanding the training results, testing the model interactively, and running batch inference on unseen data.

This lecture helps you move from model training to model validation and real-world usage.

What You Will Learn in This Lecture

By the end of this lecture, you will be able to:

• Understand fine-tuning results and metrics for a Gemini model

• Interpret training vs validation accuracy and loss

• Explore checkpoints and model versions

• Test a tuned Gemini model using Vertex AI Studio

• Perform batch inference on multiple text inputs

• Read and understand batch prediction outputs

Key Topics Covered

1. Reviewing the Fine-Tuning Status and Progress

• The tuned model Text Classifier v3 has successfully completed training

• The tuning status shows Succeeded

• Training progress is visualized using:

  • Accuracy curves

  • Loss curves

• The blue line represents training performance

• The pink line represents validation performance

This comparison helps identify overfitting or underfitting.

2. Understanding Checkpoints and Model Versions

• Multiple checkpoints are created during tuning

• Each checkpoint represents a saved model state

• A separate endpoint is deployed for each checkpoint

• This allows testing different versions of the model

• Accuracy is tracked for both:

  • Training data

  • Validation data

3. Dataset and Token Statistics

• Training dataset contains 800 labeled records

• Detailed statistics are shown for:

  • Input tokens per example

  • Output tokens per example

  • Messages per example

• You can see:

  • Minimum, maximum, mean, and median values

• This helps you understand:

  • Text length distribution

  • Model token usage

4. Exploring the Model in Vertex AI Model Registry

• The tuned model is available in the Model Registry

• Key tabs include:

  • Evaluate

  • Test

  • Batch Inference

  • Version Details

  • Lineage

• Evaluation can also be done using Generative AI Evaluation Service in Colab

5. Testing the Tuned Model in Vertex AI Studio

• The model is opened directly in Vertex AI Studio

• To get correct predictions:

  • A system instruction must be provided

  • The instruction defines classification rules:

    • 1 for positive sentiment

    • 0 for negative sentiment

• Example tests include:

  • Positive reviews → output 1

  • Negative reviews → output 0

This ensures consistent and reliable predictions.

6. Advanced Testing Options

You also explore additional configuration options such as:

• Structured output (on/off)

• Thinking budget:

  • Auto

  • Manual

  • Off

• Grounding options:

  • Google Search

  • Your own data

• Advanced parameters:

  • Temperature

  • Output token limits

  • Top-p sampling

These settings help control model behavior.

7. Running Batch Inference on Multiple Reviews

• Batch inference is created using:

  • The tuned Gemini model

  • A JSON file stored in Cloud Storage

• The batch file contains multiple text reviews

• The output is saved to a Cloud Storage folder

• Batch inference allows:

  • Large-scale predictions

  • Automated evaluation

  • Offline processing

8. Understanding Batch Inference Output

• The output file contains:

  • Input text

  • Predicted label (0 or 1)

  • Confidence information

  • Token usage details

• Each record shows:

  • Whether the sentiment is positive or negative

• This makes it easy to validate model behavior on real data

In this lecture, we focus on evaluating a fine-tuned Gemini model using the Vertex AI Generative AI Evaluation Service. After testing the model in the previous tutorial, this session helps you measure model quality, compare responses against reference answers, and understand evaluation metrics in a practical and structured way.

This lecture introduces both automatic similarity-based evaluation and model-based qualitative evaluation, which are critical for validating real-world Generative AI applications.

What You Will Learn in This Lecture

By the end of this lecture, you will be able to:

• Launch and use the Vertex AI Evaluation notebook

• Run evaluations using Colab or Vertex AI Workbench

• Understand reference-based evaluation metrics

• Use model-based pointwise and pairwise metrics

• Interpret evaluation results in a simple and practical way

• Decide whether a model is performing well enough for production

Key Topics Covered

1. Accessing the Evaluation Notebook

• The evaluation notebook is launched directly from the Evaluate tab

• You can open it in:

  • Google Colab

  • Colab Enterprise

  • Vertex AI Workbench

  • GitHub

• The notebook is provided by Google LLC and licensed under Apache License 2.0

This notebook is specifically designed for Generative AI model evaluation in Vertex AI.

2. Understanding the Purpose of the Evaluation Notebook

The notebook demonstrates how to evaluate models using the Vertex AI Python SDK for:

• First-party models (Gemini)

• Third-party models (Anthropic, OpenAI, etc.)

• Prompt engineering experiments

In this lecture, the focus is on first-party Gemini models.

3. Authentication and SDK Initialization

• The notebook requires authentication to your Google Cloud project

• You specify:

  • Project ID

  • Region (for example, us-central1)

• This ensures the evaluation runs securely within your cloud environment

4. Loading the Evaluation Dataset

• A sample evaluation dataset (Open-Orca style dataset) is used

• The dataset contains:

  • System prompts

  • Questions

  • Reference (expected) answers

• The dataset is converted into a structured format for evaluation

This dataset acts as the ground truth for comparison.

5. Reference-Based Evaluation (ROUGE-L Metric)

• The first evaluation uses ROUGE-L, a similarity-based metric

• ROUGE-L measures:

  • How much the model response overlaps with the reference answer

• Key statistics explained:

  • Mean score (average similarity)

  • Standard deviation (variation across examples)

Interpretation:

• Scores closer to 1.0 indicate strong similarity

• Lower scores mean partial or weak overlap

• This helps identify whether the model is answering correctly, even if phrasing differs

6. Understanding the Evaluation Table

Each evaluation row includes:

• System instruction

• Input question

• Reference answer

• Model response

• ROUGE-L similarity score

This allows you to:

• Inspect individual failures

• Understand why certain responses scored low

• Identify patterns in model behavior

7. Model-Based Pointwise Evaluation Metrics

In addition to similarity metrics, the lecture explores model-based qualitative metrics, such as:

• Fluency

• Coherence

• Safety

• Summarization quality

• Instruction following

• Text quality

These metrics are:

• Generated by AI models

• Based on human-like judgment

• Useful for evaluating writing quality and readability

8. Interpreting Pointwise Evaluation Results

For each response, the evaluation provides:

• A score (higher is better)

• A natural language explanation

This helps you understand:

• Why a response is fluent or not

• Whether the answer is safe and well-structured

• How well the model followed the instruction

9. Creating Custom Pointwise Metrics

The notebook also allows:

• Defining your own evaluation criteria

• Measuring custom attributes such as:

  • Linguistic acceptability

  • Domain-specific quality

• This is useful for enterprise-specific use cases

10. Pairwise Evaluation Metrics

• Pairwise evaluation compares:

  • Two model responses side-by-side

• Metrics such as:

  • Summarization quality

  • Text verbosity

• The output clearly indicates:

  • Which response is better

  • Why it is better

This is extremely helpful when:

• Comparing different prompts

• Comparing different model versions

11. Extending Evaluation to Other Models

• The same framework can be used to evaluate:

  • Third-party models hosted on Vertex AI

  • Multiple Gemini versions

• This allows fair and consistent comparison across models

12. Prompt Engineering Evaluation

The notebook also supports:

• Comparing multiple prompt templates

• Measuring how prompt changes affect quality

• Viewing experiment logs for evaluation runs

This makes evaluation a powerful tool for prompt optimization.

In this lecture, we begin our journey into AutoML for Image Data using Google Vertex AI. The primary focus of this session is image data preparation, which is one of the most important and often overlooked steps in building a successful machine learning model.

You will learn how to prepare a real-world image dataset, organize it correctly, upload it to Google Cloud Storage, and make it ready for AutoML single-label image classification in Vertex AI.

What You Will Learn in This Lecture

By the end of this lecture, you will be able to:

• Understand image AutoML options available in Vertex AI

• Select the correct image objective for your use case

• Prepare an image dataset for single-label classification

• Organize images using labels and folders

• Upload image data to Google Cloud Storage (GCS)

• Create a CSV file required by Vertex AI for image datasets

• Make your dataset fully ready for AutoML training

Key Topics Covered

1. Introduction to AutoML for Images in Vertex AI

• Overview of Vertex AI image datasets

• Available image objectives:

  • Single-label classification

  • Multi-label classification

  • Object detection

  • Image segmentation

• Why single-label classification is selected for this tutorial

This lecture focuses on classifying images into one category only.

2. Understanding the Image Dataset

• Introduction to the Rock–Paper–Scissors dataset from Kaggle

• Dataset details:

  • Total images: 2,892

  • Three classes: rock, paper, scissors

• Real-world example of a classification problem

This dataset is ideal for beginners to understand image classification concepts.

3. Downloading and Exploring the Dataset

• Downloading the dataset from Kaggle

• Extracting the ZIP file

• Understanding dataset structure:

  • Separate folders for each class

  • Images grouped by label

• Visual inspection of images to understand variations

4. Training and Testing Data Split

• Creating a training dataset:

  • 100 images per class (rock, paper, scissors)

• Using the provided test dataset as-is:

  • 124 images per class

• Importance of separating training and testing data

This ensures better model evaluation later.

5. Uploading Images to Google Cloud Storage (GCS)

• Creating folders inside a GCS bucket:

  • rock

  • paper

  • scissors

• Uploading training images to their respective folders

• Verifying successful uploads

Each folder name directly represents the label for classification.

6. Why a CSV File Is Required for AutoML Images

Vertex AI requires a CSV or JSONL file that tells the system:

• Where each image is stored (GCS path)

• What label belongs to each image

This lecture uses a CSV file, which is simpler to understand.

7. Understanding the CSV File Structure

The CSV file contains:

• Image file path (GCS URI)

• Label name

Important rules explained:

• GCS paths are case-sensitive

• Labels must:

  • Start with a letter

  • Contain only letters, numbers, or underscores

• Optional column for dataset splitting is discussed but not used

8. Generating Image Paths Using Cloud Shell

• Using Cloud Shell to list all images from each folder

• Extracting image paths automatically

• Assigning correct labels to each image

• Creating individual text files for:

  • rock

  • paper

  • scissors

• Combining all files into one CSV dataset file

This approach avoids manual errors and saves time.

9. Uploading the CSV File to GCS

• Copying the generated CSV file to the GCS bucket

• Verifying the file upload

• Downloading and validating the CSV content

Final dataset summary:

• Total records: 300

• One row per image

• Each image mapped to its correct label

In this lecture, we move to the next critical step in our AutoML image classification workflow:
creating a Vertex AI image dataset and training a machine learning model using the data prepared in the previous tutorial.

By the end of this session, you will understand how to take a fully prepared image dataset and use Vertex AI AutoML to train a single-label image classification model—without writing any machine learning code.

What You Will Learn in This Lecture

By completing this lecture, you will be able to:

• Create an image dataset in Vertex AI

• Import labeled images using a CSV file from Google Cloud Storage

• Understand dataset statistics and label distribution

• Configure AutoML training options for image classification

• Start training an AutoML image model on the cloud

• Understand compute, pricing, and training workflow in Vertex AI

Topics Covered in Detail

1. Recap of Data Preparation

We begin with a quick recap of what was completed earlier:

• Three image folders were created:

  • rock

  • paper

  • scissors

• Images were uploaded to Google Cloud Storage

• A CSV file was created containing:

  • Image file path (GCS URI)

  • Corresponding label for each image

This CSV file acts as the bridge between Cloud Storage and Vertex AI.

2. Creating an Image Dataset in Vertex AI

You will learn how to:

• Navigate to Vertex AI → Datasets

• Create a new dataset with:

  • Image as the data type

  • Single-label classification as the objective

  • Correct region selection (US Central 1)

Once created, this dataset becomes the foundation for training the model.

3. Importing Images Using Google Cloud Storage

Instead of uploading images manually, we use the CSV import method, which allows Vertex AI to:

• Read image paths directly from Cloud Storage

• Automatically assign labels from the CSV file

• Import all images in one step

Key points covered:

• Supported image formats

• Why CSV import is preferred for large datasets

• Default data split selection when no split column is provided

4. Verifying Imported Data

After import, we review:

• Total number of images

• Labeled vs unlabeled images

• Distribution across labels:

  • Rock

  • Paper

  • Scissors

• Dataset properties such as:

  • Objective type

  • Region

  • Encryption

  • Annotation sets

This step ensures your dataset is ready for training.

5. Dataset Analysis and Readiness Check

Vertex AI automatically analyzes the dataset and confirms:

• All images are labeled correctly

• Each class has sufficient data

• The dataset meets AutoML training requirements

This helps avoid training failures later.

6. Configuring AutoML Training

You will learn how to configure training options, including:

• Selecting AutoML as the training method

• Choosing Cloud deployment (not Edge)

• Training a new model version

• Naming and describing the model clearly

We also discuss when and why Edge deployment might be used.

7. Data Splitting Strategy

Training configuration includes:

• Random data split

• Default ratios:

  • 80% training

  • 10% validation

  • 10% testing

You will also learn when manual data splits might be useful.

8. Incremental Training Explained

This lecture explains:

• What incremental training is

• When it should be enabled

• Why it is disabled for this first model

This concept is important for future model improvements and cost optimization.

9. Compute and Pricing Configuration

Before starting training, we:

• Select minimum compute requirements

• Understand node-hour usage

• Choose a cost-effective training setup

This section helps you control training cost without sacrificing learning outcomes.

10. Starting the Training Process

Finally, the AutoML image classification training is started:

• Training status can be monitored in Vertex AI

• Model training runs in the background

• Expected training time is around two hours

You also learn how to track training progress from the Training section.

In this lecture, we complete the end-to-end AutoML image classification workflow by evaluating the trained model, deploying it to an endpoint, testing real-time predictions, and finally running batch inference on multiple images stored in Google Cloud Storage.

This session helps you understand how a trained AutoML image model behaves in real-world scenarios and how predictions are generated both online and in batch mode.

What You Will Learn in This Lecture

By the end of this lecture, you will be able to:

• Explore training pipeline results and completion details

• Analyze AutoML image model evaluation metrics

• Understand confidence scores, precision, recall, and confusion matrix

• Deploy an image classification model to a Vertex AI endpoint

• Perform real-time (online) image predictions

• Understand why small datasets can affect prediction accuracy

• Prepare input data for batch image inference

• Run batch inference using JSONL input files

• Analyze batch prediction outputs stored in Cloud Storage

Topics Covered in Detail

1. Training Pipeline Completion Overview

We begin by reviewing the completed training pipeline:

• Training status marked as Finished

• Total training duration of approximately 1 hour and 43 minutes

• Trained model: Rock Paper Scissors Image Classification Model

This confirms that AutoML training completed successfully.

2. Model Evaluation Results

In the Evaluate tab, we analyze model performance:

• Default confidence threshold: 0.5

• Explanation of confidence score and model certainty

• Label-wise evaluation for:

  • Rock

  • Paper

  • Scissors

• Precision and recall shown as 100%

• Dataset split:

  • Training images: 240

  • Validation images: 30

  • Test images: 30

• Confusion matrix interpretation

⚠️ Important note: Perfect evaluation metrics on a small dataset may not reflect real-world performance.

3. Deploying the Model to an Endpoint

Next, we deploy the trained model for online predictions:

• Create a new deployment endpoint

• Select region and access type

• Use default encryption settings

• Configure traffic split (100% to one model)

• Allocate compute resources

• Complete endpoint deployment

This makes the model available for real-time inference.

4. Testing Online Predictions

After deployment, we test the model by uploading images:

• Predictions return probabilities for each label

• Observed cases where:

  • Predictions are incorrect

  • Confidence scores are misleading

• Explanation of why misclassifications occur:

  • Training dataset is too small

  • Model needs more diverse data for better generalization

This highlights an important real-world ML concept:
Good evaluation scores do not always guarantee good predictions.

5. Introduction to Batch Inference

We then move to batch inference, which is used when:

• Predicting labels for many images at once

• Images are already stored in Cloud Storage

• Real-time predictions are not required

Batch inference is ideal for offline processing and large datasets.