Hands on Google cloud platform(GCP) - Data Engineer

Set up a Google Cloud account in the cloud console, claim $300 free credit, and create a new project to access services like Compute Engine and a VM instance.

Access Google Cloud Platform resources via cloud shell to manage projects, billing, and the console, and learn practical de-clawed commands for listing, configuring, and getting help.

Explore Hadoop, an open source Java framework for distributed processing of large data sets across clusters. Learn its architecture—MapReduce, YARN, HDFS, and common utilities—and its evolution into the Hadoop ecosystem.

Explore the Hadoop ecosystem on Google Cloud Platform, with Kafka streaming data, and understand each component's role in a managed GCP deployment.

Explore Hadoop on Google Cloud Platform and understand why it powers big data. Learn how Hadoop distributes data and work across machines, supports fault tolerance, remains open source and Java-based.

Compare the Hadoop ecosystem with GCP equivalent, detailing HDFS with name node and data node, MapReduce batch processing, and GCP tools like Dataproc, Spark, and Dataflow for big data analytics.

Explore three cloud computing options for hosting websites, storing data, and enabling machine learning on Google Cloud Platform, including App Engine and Compute Engine.

Explore preemptible VMs, a cheaper option on Google Cloud Platform that can be terminated after 24 hours with a 30-second notice. They are ideal for short jobs.

Explore Google App Engine, comparing standard and VM-based options, and learn how a managed, scalable platform offers auto scaling, traffic splitting, zero downtime, and minimal operational overhead for websites.

Create a Google Cloud Platform data engineer lab VM with Compute Engine, name it, select region and zone, review pricing and firewall options, then connect via SSH and update software.

Edit and manage compute engine VMs by changing machine types, stopping instances, and adding disks, while configuring access scopes and service accounts. Clean up by deleting unused instances after use.

Create and manage Google Cloud VM instances from the command line, set a default zone, apply labels, and script multiple instances with consistent configurations.

Explore Google Cloud Storage fundamentals, including buckets, storage classes (multi-regional, regional, nearline, coldline), and storage types (standard, SSD, local SSD), and map data workloads to cloud storage tools for analytics.

Explore cloud storage concepts including buckets, globally unique bucket names, and storage classes like multi-regional and regional, with nearline options, pricing, redundancy, and data transfer via the transfer service.

Explore Google Cloud Datastore: a scalable store for highly structured or hierarchical data, offering fast key lookups with entities and properties, and indexing options with trade-offs in filtering and joins.

Learn how Cloud SQL enables relational databases for OLTP with ACID properties and horizontal scaling. Explore first and second generation instances, Cloud SQL proxy, replication, and security features.

Learn how Cloud Spanner enables horizontal scaling with shards and instances, ensures high availability and strong consistency, and uses primary keys and secondary indices to manage relational data and transactions.

Learn how to set up a Cloud Spanner instance, choose region and nodes, create a database, and define schemas and tables with interleaved, parent-child relationships and primary keys.

Create a storage bucket with a globally unique name, choose a storage class based on global access patterns (nearline or coldline), and easily upload files.

Learn how to create and configure a Cloud SQL instance on Google Cloud Platform, choosing MySQL or PostgreSQL, setting credentials, and enabling backups and binary logging.

Explore Bigtable as a scalable columnar database designed for fast sequence scans of sparse data, with dynamic column additions and column families, enabling single reads of related attributes.

Explore CRUD operations in Bigtable, learn about its limitations, including no multi-row operations, no indexes or constraints, and row-level atomicity; understand column families, timestamps, and dynamic schema.

Explore Datalab by using interactive notebooks (Jupyter/Anaconda) in containers, run code via a web browser, auto-save and clone notebooks, and connect to Google Cloud resources.

Explore the pub/sub streaming pattern in cloud data engineering: publishers publish messages to topics with data and attributes, subscribers pull or push via endpoints, using acknowledgments and storage.

Explore Google Cloud Platform Dataflow pipelines that transform data from sources to sinks using transforms and Apache Beam, with PCollections, immutability, and side inputs.

Learn BigQuery basics: manage datasets, tables, and views within projects; load data from Cloud Storage as external tables; query via interactive or batch modes with partitions and view access.

Learn to load csv data into BigQuery by creating a dataset, uploading a csv file, and configuring a table with auto-detected or manual schema.

Learn to write a simple query to retrieve data from a table, apply a 500-row limit, view results with next and previous navigation, and download the data.