Google Cloud Data Engineer

Explore the five main Google Cloud compute services—Compute Engine, App Engine, Google Kubernetes Engine, Cloud Run, and Cloud Functions—and compare their use cases, configurations, and pricing.

Create your first Google Cloud VM, choose region and machine type, install Debian Linux, enable SSH, and set up Apache to host a simple web page.

Explore Google App Engine, a fully managed serverless platform for deploying and hosting web applications at scale, supporting multiple languages and environments, with hands-on deployment using Cloud Console and gcloud.

Master Google Cloud networking essentials, including IP addresses, VPC, subnets, firewalls, and NAT gateways, and explore VPC peering and shared VPC for secure interconnectivity.

Explore Google Cloud storage options, from object storage to block storage. Review relational, NoSQL, data warehousing, and in-memory options like Cloud SQL, Cloud Spanner, Datastore, Firestore, Bigtable, BigQuery, Memorystore.

Create and manage Google Cloud Storage buckets, configure region and storage class, upload objects, and enable versioning, retention policies, and lifecycle rules to protect data.

Learn to create and manage Google Cloud Storage buckets, choose storage classes and regions, and use object versioning, retention policies, and lifecycle rules to protect and automate data.

Learn to manage Google Compute Engine disks, including boot and data disks, attach and mount them to a Linux VM, and use incremental snapshots for backups and cross-region restoration.

Create and manage a daily snapshot schedule for a disk, set daily 7 to 8 a.m. UTC, attach it to disks, monitor incremental snapshots, and restore the latest snapshot.

Mount a Google Cloud Storage bucket as a file system on a GCP VM using GCS FUSE, with setup, cloud net access, and live file operations.

Launch and configure a Cloud SQL MySQL instance in Google Cloud, including enabling the Cloud SQL Admin API, setting region, storage, backups, and connecting via Cloud Shell and MySQL Workbench.

Migrate on-prem MySQL data to Cloud SQL by exporting a SQL dump from on-prem, uploading to a Cloud Storage bucket, and importing into Cloud SQL.

Explore Google Cloud Spanner, a fully managed relational database that provides transactional consistency at global scale with automatic synchronous replication across three zones, contrasting it with Cloud SQL.

Explore Cloud Spanner, a fully managed relational database in GCP, and compare it with Cloud SQL, highlighting regional and multi-regional configurations, automatic synchronous replication, and global transactional consistency.

Explore how to create a Cloud Spanner change stream and use a streaming Dataflow job to load change stream data into BigQuery for near real-time change data capture.

Explore Cloud Bigtable, a fully managed NoSQL database, and its use for time series, financial, and IoT data, with the CBT command line tool for setup.

Create a cloud Bigtable instance, configure a demo EMP table with two column families, and use the CBT CLI in Cloud Shell to read, write, and backup/restore data.

Explore Google Cloud BigQuery, a managed, serverless data warehouse for analytics and machine learning, with data sets, tables, and jobs, plus columnar storage and encryption with KMS or customer-managed keys.

Create a dataset and table in BigQuery, upload a CSV, run sample queries, understand caching and cost based on data processed, and optimize by selecting specific columns.

Explore how partitioning and clustering in BigQuery optimize performance by date-based partitioning, reducing data scanned and costs, as shown on a large tech forum table.

Learn how to control access in BigQuery by sharing a dataset, removing dataset access, and granting table-level permissions with BigQuery Data Viewer, then verify access with queries.

Explore BigQuery external data sources and federated queries, creating external tables from Cloud Storage, S3, or Azure Blob Storage, and querying Cloud Spanner via the external query function.

Discover how to restore deleted or updated data in BigQuery with time travel, querying past states within a dataset at the dataset level, which supports a seven day default window.

Discover how BigQuery stores data column by column and uses a streaming buffer for inserts. Block updates or deletes on streaming data until it moves to permanent storage.

Discover how Gemini AI in BigQuery (formerly Duet AI) enables prompt-based query generation, explanation, and auto-completion, with practical steps to enable data access and run queries.

Explore Google Cloud data processing services for ETL, from ingestion with pub/sub and storage to transformation and loading into BigQuery and other data stores.

Explore Google Cloud Pub/Sub, a reliable asynchronous publish/subscribe messaging service for real-time data processing, comparable to Kafka or RabbitMQ, and used by Google products to ingest and distribute streaming data.

Explore Google Cloud Dataflow, a serverless unified stream and batch platform. Create templates to move spanner data to csv in GCS, stream pub sub to BigQuery with change data capture.

Create a batch dataflow job to compress CSV files in a Google Cloud Storage bucket using a utility template, outputting gzip files into a zip folder and a failure folder.

Learn to load a cloud storage csv into BigQuery using a batch dataflow template, with a JavaScript UDF and a JSON schema to map columns.

Extract data from cloud spanner to a csv file in google cloud storage using a data flow inbuilt template.

Create a Cloud Spanner change stream and a streaming Dataflow job to load changes into BigQuery, enabling near real-time change data capture for inserts, updates, and deletes.

Learn to enable change data capture with Cloud Spanner by creating a change stream and feeding near real time inserts, updates, and deletes into BigQuery via a Dataflow streaming job.

Google Cloud Data Fusion is a fully managed, cloud-native data integration service with a code-free, drag-and-drop UI that runs on Dataproc and supports 100+ plugins and Google Cloud connectors.

Build a batch data pipeline with Cloud Data Fusion, transforming a csv in cloud storage, filtering offline sales, masking total profit, and loading to the BigQuery table Data Fusion sync.

Explore Google Cloud Composer, a fully managed Apache Airflow service, to orchestrate DAG-based workflows with tasks, dependencies, and operators across Google Cloud, including VM backups and data pipelines.

Create a Cloud Composer environment on a GKE cluster with Apache Airflow, provisioning storage, logging, and monitoring. Use a service account with permissions and enable the Cloud Composer API.

Explore the cloud composer architecture, including cloud sql airflow database, storage, tenant project isolation, a kubernetes engine cluster, and gcs bucket file system via gcs fuse with monitoring and logging.

Create a cloud composer dag by writing a python dag and uploading it to the dex folder for airflow ui visibility, orchestrating tasks with dag concepts.

Restart Google Cloud Composer services by rebooting the web server and scheduler on a GKE cluster with kubectl rollout restart, and use a beta command to restart the web server.

Explore Google cloud ai/ml services—pre-trained models, autoML, and custom training—and implement vision, speech, natural language, translation, and video intelligence APIs via cloud console or cloud shell.

Explore Google's vision API as a pre-trained model, performing face detection, label and text extraction, and safe content checks, using cloud shell and a Python client to process images.

Explore how the natural language processing api derives insights from unstructured text, including sentiment analysis, entity detection, and syntax, with examples from customer reviews.

Use the Google Cloud Video Intelligence API to detect explicit content in a video uploaded to cloud storage, validating results frame by frame via cloud shell and gcloud.

Explore generative AI studio in Google Cloud's Vertex AI to prototype and test chatbots using palm API or OpenAI APIs, customize with context, and explore language, vision, and speech features.

Learn to build a custom machine learning model with Google AutoML and Vertex AI for image classification. Train with labeled data, deploy to an endpoint, and evaluate model accuracy.