Google Cloud Professional Data Engineer Course [2019 Update]

Explore compute engine by launching virtual machines, learning about zones and regions, machine types, disks, snapshots, and images. Spin up a vm via the gcp console and connect with ssh.

Explore compute engine by selecting VM machine types—micro, small standard, high memory, or high CPU—customizing memory, CPU, and boot disk options, and using preemptible and permanent instances with snapshots efficiently.

Select a project and navigate to compute engine to create a new vm instance, choosing zone, machine type, memory, boot disk, and enabling http/https traffic.

Compare VM configurations to balance cost and performance, noting disk type and size influence pricing, and learn to access VMs via SSH from Mac and Windows, using SSDs for speed.

Generate ssh keys on mac, upload the public key to Google Cloud, and connect to your VM with ssh using your private key and the instance's external IP.

Use Windows PuTTY to SSH a Google Cloud VM by generating RSA keys, saving the private key, adding the public key to GCE, and connecting.

Explore identity access management in GCP, including member types, predefined and custom roles, and how Google accounts, service accounts, and Google groups receive and customize permissions.

Explain iam primitive roles—owner, editor, viewer—and illustrate policy inheritance across the google cloud resource hierarchy (organization, folders, projects) to control downstream access.

Learn how every resource belongs to a project and how billing accounts attach to projects, with daily budgets, cross-project data transfer charges, and charges by the initiator of the transfer.

Explore IAM and billing in the GCP console by linking two projects to a billing account and understanding ownership, editor, and viewer roles across project and resources like BigQuery.

Explore gcp storage solutions and databases, including cloud storage, bigtable, bigquery, datastore, spanner, memorystore, and analytics concepts like oltp vs olap and strong vs eventual consistency to guide selection.

Explore BigQuery, a serverless Google Cloud data warehouse for data analytics, detailing storage, streaming vs batch costs, partition and wildcard tables, authorized views, and SQL options to reduce query expenses.

Master partition tables in BigQuery using the underscore partition time pseudo column to filter by date, and employ wildcard tables with table star and underscore table suffix syntax for unions.

Explore BigQuery's legacy versus standard SQL, and learn cost-saving techniques like preview mode, partitioning, and avoiding select star to estimate and reduce query costs.

Create and use temporary tables in BigQuery to cache query results for about 24 hours, not visible or shareable, saved in a dataset, with formats including csv and newline-delimited json.

Practice BigQuery with the NOAA climate public dataset, using wildcard tables from 2000 to 2019, selecting specific columns to save cost, and saving results to a new destination table.

Create an authorized view in the share views dataset to restrict a data analyst to select weather data tables, then grant access via authorized views and IAM.

Explore Cloud DataLab, built on Jupyter Notebook (formerly IPython), to process data with Python and JavaScript on a Google Compute Engine VM, connected to BigQuery and Google Cloud Storage.

Enable cloud source repo API, create a DataLab VM, access BigQuery with pandas dataframes in a notebook, run 2000–2017 temperature query, plot a bar chart, and shut down the VM.

Explore Google Cloud Data Studio, GCP's dashboarding solution for live dashboards from BigQuery, Cloud SQL, MySQL, Google Sheets, and Analytics, with proper sharing controls.

Learn to access Google Data Studio, connect BigQuery weather dataset, compute average temperature 2000–2017, build a time series chart, customize axes, and share reports with view or edit permissions.

Explore Google Cloud Pub/Sub, a fully managed real-time messaging service that decouples publishers and subscribers via topics and subscriptions, with at-least-once delivery and support for data streaming and reliability improvements.

Balance workloads by evenly distributing messages across workers, enabling reliable data streaming from IoT sensors to back-end services, while multi-zone deployments ensure durability, low latency, and encryption.

Explore Hadoop fundamentals and how Dataproc integrates with the Hdfs to store and process big data, highlighting Hdfs, Hive, Pig, Oozie, and Sqoop for the Google Cloud data engineer exam.

Explore Hive for SQL-like queries on data in HDFS, translated to MapReduce jobs, and Pig for scripting transformations in Pig Latin; learn Oozie for scheduling Hadoop ETL workflows.

Explore Cloud Dataproc, a fully managed GCP service that runs Apache Hadoop clusters, with per-second VM billing, fast processing, easy scaling, and migration of existing Hadoop workloads without rewriting code.

Explore Apache Spark, an in-memory distributed data processing engine that integrates with the Hadoop ecosystem, runs on Hadoop clusters, is often faster than MapReduce, and supports transferring workloads to Dataproc.

Cloud Dataflow is a fully managed, serverless service that transforms streaming or batch data with autoscaling, enabling BigQuery analysis by converting Pub/Sub and Cloud Storage inputs into table formats.

Discover the four core concepts of cloud dataflow: P collection, transforms, and input-output sources and sinks, and how a pipeline orchestrates data from pub sub to BigQuery.

Deploy your pipeline code to cloud dataflow managed service to run as a dataflow job on compute engine; monitor with dataflow interface or command line interface, and update streaming jobs.

Deploy and test a dataflow job end-to-end from Cloud Shell using Python Apache Beam wordcount, then deploy to Dataflow with a Cloud Storage bucket as staging area.