Spark 3 on Google Cloud Platform-Beginner to Advanced Level

Explore Google Cloud Dataproc, a managed Spark and Hadoop service on GCP with preinstalled components like Hive and Python, enabling quick cluster setup for batch and real-time processing.

Discover how Spark runs on cloud with Dataproc, using Jupyter notebooks or Spark Submit, via a Spark Session on the master and a resource manager like Yarn.

Explain the differences between data warehouses and data lakes, including structure, sources, raw data, and ETL/ELT-driven analytics workflows.

Explore how Spark integrates with data lakes and warehouses, connects to databases and NoSQL sources, and handles batch and streaming data from Kafka, PubSub, and Kinesis.

Explore Spark APIs from RDDs to DataFrame and Spark SQL, highlighting the move to structured data processing and the capabilities for batch, streaming, and machine learning on Spark 3.

Explore Spark 3's adaptive query execution and GPU acceleration, learn about pandas UDFs in PySpark, and see improvements in SQL compatibility and window functions.

Learn Spark in 2023 to access growing cloud demand, a large community, and abundant job opportunities, while Spark's versatility integrates with BigQuery, Snowflake, Redshift, Kafka, and Pubsub.

Create a Dataproc cluster, run Spark scripts in Jupyter notebooks, transform data with DataFrame API, apply aggregations and window functions, and write results to a sink; submit jobs via gcloud.

Learn to create a dataproc cluster by selecting region and zone, configuring a standard cluster with one master and two workers, and enabling component gateway and Jupyter Notebook.

Explore the dataproc cluster monitoring tabs, understand yarn memory and pending memory, CPU utilization, view running jobs, manage VM instances and configurations, and launch Jupyter Lab for spark development.

Learn to create a PySpark dataframe from a CSV in a Google Cloud Storage bucket, then select columns, print the schema, and apply filters with show.

Learn to typecast string columns to integer and timestamp types in PySpark, and extract year, month, and day from timestamp columns for robust analytics.

Explore Spark dataframe aggregations by deriving post type from tags, casting view counts, and computing total views and total row count by post type.

Explore Spark transformations and actions, showing how reading a CSV from GCS creates new dataframes through filtering, selection, and aggregation, with lazy evaluation delaying execution until an action.

Master advanced window function techniques in Spark 3 on Google Cloud Platform. Compute row numbers and ranks by creation year for post types using PySpark dataframes and total views.

Learn to compute rolling sums and rolling averages of monthly views in PySpark using dataframe window transformations, partitioned by post type and creation year and month to produce cumulative totals.

Learn to write Spark job outputs to sinks such as BigQuery and GCS, configure jars for BigQuery, and save final results as parquet or json with proper staging buckets.

Submit spark jobs to dataproc clusters with spark-submit after notebook development, using a modular pyspark script with read data, df basic transforms, and df aggregations that write to BigQuery.

Explore spark sql, a spark module for structured data processing built on dataframes. Run sql queries on registered dataframes using select, where, group by, and window functions with distributed execution.

Highlight the two Spark SQL table types: temp tables for session-bound in-memory data and permanent tables with managed or unmanaged options, including Spark Warehouse and Metastore storage.

Upload the Bikeshare trips parquet file and read it with spark.read.parquet. Register a temp table to run SQL queries, then create permanent and external tables, show tables, and drop them.

learn spark sql aggregations and window functions on a parquet bikeshare dataset, rank monthly top three stations by total trips, and save results as a permanent table.

Learn how to perform joins in PySpark using DataFrame API and Spark SQL, including inner, left, and full outer joins, with proper aliasing and handling ambiguous columns.

Spark dataframes partition data across a Dataproc cluster to improve parallelism. View partition counts with numPartitions and learn that file counts don’t determine partitions, then explore how to control them.

Explore how repartitioning and coalesce adjust dataframe partitions, learn about shuffling and data movement, and write repartitioned data to a GCS bucket as multiple parquet files.

Understand how partitions trigger shuffle joins in Spark when joining two dataframes on id keys, shuffling data across workers to enable a local join, a potentially expensive operation.

Explore creating and using Spark UDFs in the DataFrame API and Spark SQL, including a clean_string example with regex and guidance on when to rely on built-ins.

Uncover how the Spark Catalyst optimizer analyzes code and resolves names. Follow four optimization stages—logical optimization with predicate pushdowns and data pruning, then code generation—to accelerate Spark SQL and dataframes.

Discover how caching and persisting in Spark store intermediate results in memory or disk, compare memory only, memory and disk, and disk only, and decide by dataframe size.

Discover how autoscaling in spark and dataproc automatically adjusts CPU and memory to balance performance and cost. Learn to prevent overprovisioning and underprovisioning by applying scaling policies to clusters.

Learn to create and apply a Dataproc autoscaling policy via YAML, configuring worker counts, weights, scaling algorithm, and apply it with gcloud to a cluster.

Develop your spark script in Jupyter notebooks and execute with spark submit using Dataproc workflow templates to automate ephemeral or existing clusters.

Execute a PySpark job that performs Spark aggregations, reads a CSV file from a GCS bucket, and writes results back to GCS via a managed Dataproc workflow.

Automate Spark job execution on a Dataproc cluster by scheduling workflows with Cloud Scheduler, including creating a PySpark trigger role and service account to trigger the workflow.

Enable recovery from failures by checkpointing in pyspark, saving the state of a distributed computation to a fault-tolerant storage like hdfs on dataproc or gcs on gcp.

Learn how broadcast joins optimize Spark by sending the smaller dataframe to each worker, reducing shuffles and network overhead, and know when to apply it.

Learn to set up alerting policies for Spark jobs on Dataproc, and receive alerts via email, SMS, webhooks, or Slack, while configuring metrics, filters, rolling windows, and thresholds.

Explore batch processing with PySpark by extracting data from a MySQL database on GCP Cloud SQL, transforming it, and loading into a GCS sink via workflows.

Set up a MySQL 8.0 Cloud SQL instance on Google Cloud, name it retail-e-commerce, enable private and public IP access, and prepare development in us-central1 for Spark job connectivity.

Connect to a Google Cloud MySQL instance, create the e-commerce_db database with two tables, and ingest orders and order_items data from CSV files for PySpark use.

Create a dataproc cluster to install the MySQL connector for Python via an initialization script uploaded to a GCS bucket, and stop clusters on trial accounts to preserve notebooks.

Connect to MySQL database with PySpark script, read into pandas dataframe, convert to a spark dataframe, and export to parquet dataset partitioned by createdat date in GCS bucket via spark-submit.

Transform and analyze order data with pyspark by extracting from mysql to gcs, performing aggregations, joining with products, and producing top products and monthly brand running totals in parquet.

Upload PySpark scripts to a GCS bucket and chain a MySQL extraction with a PySpark ETL transformation in a Dataproc workflow template to run an end-to-end pipeline.

Explore real-time data processing with spark by implementing streaming pipelines in pyspark. Get an overview of pub sub lite and preview a real-time product recommendation system project.

Explore Google Cloud Pub/Sub and Pub/Sub Lite, including topics, subscriptions, and Spark structured streaming, with Kafka-like partitions and lower-cost single-zone deployment.

Explain tumbling windows as fixed-size, non-overlapping intervals where each event belongs to a single window. Use stateless aggregations to count events in five-minute windows.

Explore watermarking in Spark streaming, distinguishing event and publishing timestamps, handling late data with the state store, and safely deleting outdated windows to avoid data loss.

Explain how tumbling windows differ from sliding windows by showing non-overlapping ten-minute windows and overlapping five-minute sliding windows used to detect users spending more than five minutes on a site.

Create a PubSub Lite reservation by enabling the API, provisioning a light reservation in us central1 with throughput capacity, linking a topic to it, then creating a light subscription.

Publish 100 website event records from events.json to a pub sub lite topic with a Python script, then run a Spark streaming job to read and display data from subscription.

Implement tumbling window aggregations with Spark streaming on Google Cloud Platform, publishing data to pub sub, defining a JSON schema, and producing two-minute window counts with complete output.

Learn to implement tumbling windows with a two-minute watermark in Spark on Google Cloud, publishing late data to Pub/Sub and computing per-minute traffic-source counts.

Implement sliding windows to detect users browsing longer than five minutes with a spark job on a ten-minute window and five-minute slide, publishing to pub sub lite.

Explore joining streaming dataframes with static dataframes, using Pub/Sub or Kafka with a BigQuery table to determine discounts and deliver results in near real time via GCS and Cloud Functions.

Join a streaming dataframe with a static dataframe from user_product_discounts.csv using a pub/sub topic, and write the inner join to a GCS bucket every ten seconds.

Join two streaming dataframes on product ID to count unique buyers in a 30-second sliding window, using two-minute watermarking, state store, and console output.

Learn watermarking in streaming joins with Spark on Google Cloud, write outputs to GCS, and tune window duration and trigger time to observe watermarking.

Implement a near real-time product recommendation pipeline using Spark structured streaming with Pub/Sub Lite and an ALS model trained with PySpark on historical data.

Train a collaborative filtering model on Google Cloud Platform using Spark ML ALS, with user and product indexers, and save the artifact to a GCS bucket. Data prep and training.

Walk through Spark code to load trained model from the gcs bucket, generate ten product recommendations per user, perform a left join with streaming events, and publish results to pub/sub.

Submit the spark job from a Jupyter notebook to generate real-time product recommendations, load the trained model from the GCS bucket, and view results in the console.

Learn interview best practices for Spark on cloud roles: stay honest about what you know, focus on fundamentals and relevant projects, and tailor expectations for startups and SMEs.

Explain the end-to-end Spark pipeline for batch data on Google Cloud Dataproc, from defining the objective to enriching transactional and clickstream data for daily and hourly reports.

Explain deploying Spark applications on Dataproc in client or cluster mode, including default behavior, running from Jupyter or gcloud submit, and the roles of executors and executor cores.

Explore spark fault tolerance, including automatic task rescheduling after an executor failure in a dataproc cluster, and understand predicate pushdown with parquet files on GCS; compare caching, persisting, and checkpointing.

Explore real-time streaming in Spark structured streaming on Google Cloud Platform, handling late arriving data with watermarking and state stores, and compare processing time and event time triggers.

Master Spark Structured Streaming performance optimization by tuning windowing and watermarking, batch size, and data serialization with Avro and Protobuf, plus monitoring, fine-tuning with alerting, and a project walkthrough.