GCP: Complete Google Data Engineer and Cloud Architect Guide

The Google Cloud for ML with TensorFlow, Big Data with Managed Hadoop

Created byLoony Corn

Last updated 7/2018

English

German [Auto],English [Auto],

What you'll learn

Deploy Managed Hadoop apps on the Google Cloud
Build deep learning models on the cloud using TensorFlow
Make informed decisions about Containers, VMs and AppEngine
Use big data technologies such as BigTable, Dataflow, Apache Beam and Pub/Sub

Course content

21 sections • 226 lectures • 27h 51m total length

You, This Course and Us2:01
Course Materials0:11

About this section0:15
Compute Options9:16
Google Compute Engine (GCE)7:38
Explore Google Compute Engine as an infra as a service that delivers fine-grained load balancing, autoscaling, and configurable virtual machine instances for scalable web apps.
Lab: Creating a VM Instance5:59
More GCE8:12
Lab: Editing a VM Instance4:45
Lab: Creating a VM Instance Using The Command Line4:43
Lab: Creating And Attaching A Persistent Disk4:00
Google Container Engine - Kubernetes (GKE)10:33
More GKE9:54
Lab: Creating A Kubernetes Cluster And Deploying A Wordpress Container6:55
App Engine6:48
Compare the app engine standard and flexible environments, and learn when OS customization is possible, using containers versus virtual machines in a serverless platform.
Contrasting App Engine, Compute Engine and Container Engine6:02
Lab: Deploy And Run An App Engine App7:29
Create a simple python app with webapp2, configure app.yaml, test locally, and deploy to production on app engine's serverless platform without managing virtual machines.
Compute

About this section0:11
Storage Options9:48
Quick Take13:41
Cloud Storage10:37
Lab: Working With Cloud Storage Buckets5:25
Learn to create and manage Google Cloud Storage buckets, choose multi-regional, regional, nearline, or coldline storage classes for emergencies, and understand per-bucket and per-object storage class behavior.
Lab: Bucket And Object Permissions3:52
Lab: Life cycle Management On Buckets5:06
Learn to manage lifecycle and permissions for Google Cloud storage buckets using gsutil to create buckets, copy objects, and apply a json lifecycle policy that deletes files after 30 days.
Fix for AccessDeniedException: 403 Insufficient Permission0:30
Lab: Running A Program On a VM Instance And Storing Results on Cloud Storage7:09
Transfer Service5:06
Transfer service enables importing data into cloud storage from AWS S3, HTTP(S), on-premise files via gsutil, or between cloud buckets, with recurring transfers and synchronization to keep data in sync.
Lab: Migrating Data Using The Transfer Service5:32
gcloud init0:05
Lab: Cloud Storage ACLs and API access with Service Account7:49
Lab: Cloud Storage Customer-Supplied Encryption Keys and Life-Cycle Management9:27
Lab: Cloud Storage Versioning, Directory Sync8:41

About this section0:15
Cloud SQL7:40
Enable secure access to Cloud SQL by using a Cloud SQL proxy to establish a secure TCP tunnel, avoiding IP whitelisting and simplifying credentials and connection management.
Lab: Creating A Cloud SQL Instance7:54
Lab: Running Commands On Cloud SQL Instance6:31
Lab: Bulk Loading Data Into Cloud SQL Tables9:09
Cloud Spanner7:25
More Cloud Spanner9:18
Explore how Cloud Spanner delivers strong serializable transactions with commit timestamps, while avoiding hotspotting with hashed keys, and using splits and indices to balance performance and query flexibility.
Lab: Working With Cloud Spanner6:49
Set up a Cloud Spanner instance, choose a region, and provision production-ready nodes. Follow guidelines to use at least three nodes and keep CPU utilization under 75% for best performance.
Important! Delete unused GCP projects/instances0:21
Just wanted to send along an important note for anyone learning a cloud technology like GCP - please be sure to delete your projects, instances and in general to free up your resources after you are done using them. Resources like BigTable, Cloud Spanner are pretty expensive - if you happen to create one, then forget to free it up, you could be hit with real sticker shock when you get your next invoice.

Just something important to keep in mind if you are new to using pay-as-you-go technologies:-)

About this section0:15
BigTable Intro7:57
Columnar Store8:12
Denormalised9:02
Column Families8:09
BigTable Performance13:19
Getting the HBase Prompt0:15
Lab: BigTable demo7:39
Important! Delete unused GCP projects/instances0:21
An important note for anyone learning a cloud technology like GCP - please be sure to delete your projects, instances and in general to free up your resources after you are done using them. Resources like BigTable, Cloud Spanner are pretty expensive - if you happen to create one, then forget to free it up, you could be hit with real sticker shock when you get your next invoice.

Just something important to keep in mind if you are new to using pay-as-you-go technologies:-)

About this section0:15
BigQuery Intro11:03
BigQuery Advanced9:59
Lab: Loading CSV Data Into Big Query9:03
Lab: Running Queries On Big Query5:26
Lab: Loading JSON Data With Nested Tables7:28
Load newline-delimited json into BigQuery, using auto-detect or a json schema, and access nested fields with the dot operator, including cities lived.
Lab: Public Datasets In Big Query8:16
Lab: Using Big Query Via The Command Line7:45
Lab: Aggregations And Conditionals In Aggregations9:51
Lab: Subqueries And Joins5:44
Lab: Regular Expressions In Legacy SQL5:36
Lab: Using The With Statement For SubQueries10:45

About this section0:04
Data Flow Intro11:04
Illustrates data flow as a directed acyclic graph of transformations in a pipeline, moving data from sources to sinks like BigQuery, with backends such as Spark or Flink via runner.
Apache Beam3:42
Lab: Running A Python Data flow Program12:56
Lab: Running A Java Data flow Program13:42
Lab: Implementing Word Count In Dataflow Java11:17
Lab: Executing The Word Count Dataflow4:37
Lab: Executing MapReduce In Dataflow In Python9:49
Execute a mapreduce-style dataflow pipeline in Python with Apache Beam to identify the top N Java packages by import counts, run locally or on Google Cloud Dataflow.
Lab: Executing MapReduce In Dataflow In Java6:08
Lab: Dataflow With Big Query As Source And Side Inputs15:50
Lab: Dataflow With Big Query As Source And Side Inputs 26:28
Explore running a dataflow pipeline on Google Cloud from BigQuery input to Cloud Storage, monitor autoscaling under quota limits, and interpret the execution graph with parallel pipelines and side inputs.

Requirements

Basic understanding of technology - superficial exposure to Hadoop is enough

Description

This course is a really comprehensive guide to the Google Cloud Platform - it has ~25 hours of content and ~60 demos.

The Google Cloud Platform is not currently the most popular cloud offering out there - that's AWS of course - but it is possibly the best cloud offering for high-end machine learning applications. That's because TensorFlow, the super-popular deep learning technology is also from Google.

What's Included:

Compute and Storage - AppEngine, Container Enginer (aka Kubernetes) and Compute Engine
Big Data and Managed Hadoop - Dataproc, Dataflow, BigTable, BigQuery, Pub/Sub
TensorFlow on the Cloud - what neural networks and deep learning really are, how neurons work and how neural networks are trained.
DevOps stuff - StackDriver logging, monitoring, cloud deployment manager
Security - Identity and Access Management, Identity-Aware proxying, OAuth, API Keys, service accounts
Networking - Virtual Private Clouds, shared VPCs, Load balancing at the network, transport and HTTP layer; VPN, Cloud Interconnect and CDN Interconnect
Hadoop Foundations: A quick look at the open-source cousins (Hadoop, Spark, Pig, Hive and HBase)

Who this course is for:

Yep! Anyone looking to use the Google Cloud Platform in their organizations
Yep! Any one who is interesting in architecting compute, networking, loading balancing and other solutions using the GCP
Yep! Any one who wants to deploy serverless analytics and big data solutions on the Google Cloud
Yep! Anyone looking to build TensorFlow models and deploy them on the cloud

GCP: Complete Google Data Engineer and Cloud Architect Guide

What you'll learn

Explore related topics

Course content

You, This Course and Us2 lectures • 2min

Introduction4 lectures • 24min

Compute14 lectures • 1hr 32min

Storage15 lectures • 1hr 33min

Cloud SQL, Cloud Spanner ~ OLTP ~ RDBMS9 lectures • 55min

Hadoop Pre-reqs and Context1 lecture • 1min

BigTable ~ HBase = Columnar Store9 lectures • 55min

Datastore ~ Document Database3 lectures • 21min

BigQuery ~ Hive ~ OLAP12 lectures • 1hr 31min

Dataflow ~ Apache Beam11 lectures • 1hr 36min

Requirements

Description

Who this course is for: