Learn Practical Apache Beam in Java | BigData framework

Create a PCollection from a file system using Apache Beam in Java, reading an input CSV and writing a single output CSV, with transforms and pipeline execution.

Derive a PCollection from an in-memory Java object by creating a customer entity, converting it to strings with map, and writing to csv using TextIO.

Learn how to replace hardcoded file paths with custom beam pipeline options, enabling command line input for input, output, and extension, and run the jar via maven with arguments.

Learn how a p transform represents a data processing step that converts a p collection into a new p collection, using the apply method for elementwise and aggregation transformations.

Learn map elements in apache beam to perform 1-to-1 transformations on a pcollection, using descriptors to convert names from lowercase to uppercase and write results to csv.

Explore ParDo in Apache Beam as an element-wise transform with 1-to-1, 1-to-0, and 1-to-many outputs, using a DoFn to filter Los Angeles customers and write CSV with headers.

Demonstrates using Apache Beam's filter transform API in Java, implementing a serializable function to filter input strings by Los Angeles and writing results to customer_filter_output.csv from customer_pardot.csv.

Explore the flatten transformation in Apache Beam by merging multiple PCollection objects into a single collection using the flatten operation.

Partition a beam collection into city-based outputs with a custom partition class in Java, demonstrating three partitions for Los Angeles, Phoenix, and New York and writing results to disk.

Explore side inputs in Apache Beam by using a side input map to filter customers who never return products, demonstrated with reading order and return csv files in Java.

Remove duplicate records from a collection using the distinct class in Apache Beam for Java, processing a csv to produce a unique list of customers.

Learn to count the total number of records in a PCollection using count.globally in Apache Beam Java, then print the result with a ParDo and an anonymous DoFn.

Learn how the group by key transformation aggregates amounts by customer id. Build key-value pairs from a CSV, group values by key into an iterable, and sum them.

Implement a group by key transformation in Apache Beam with Java, convert input to KV pairs, sum amounts by customer id, and write the results to output.

Learn to implement an inner join in Apache Beam using CoGroupByKey in Java to join a user orders dataset with a user details dataset on the user ID.

Explore the right outer join, which returns all records from the right table and matching left records. See nulls where no left match, demonstrated with a right join example.

establish a connection between s3 and apache beam in java by creating a bucket, an input folder, uploading a csv, and configuring public read access.

Learn how to connect AWS S3 with Apache Beam in Java, configure access and secret keys, read a CSV from S3 using the S3 protocol, and print elements with ParDo.

Learn how to read csv data with beam, convert to generic records using a defined schema, and write to parquet files with parquet io in java.

Demonstrate reading a parquet file with Apache Beam in Java, define a print element function extending the simple function class, apply the schema, and print each element.

Learn to connect MySQL with Apache Beam using JDBC IO by configuring the data source and reading the product_info table with a prepared statement, then export to jdbc_output.csv.

Learn how to integrate MongoDB with Apache Beam in Java by building a pipeline that reads a CSV file and writes JSON documents to the training.user collection.

Learn how to integrate Apache Beam with HDFS by provisioning an AWS EMR Hadoop cluster, loading data, and running a Beam pipeline with the direct runner.

Build a streaming ETL pipeline that processes real-time IoT data, counts events where temperature exceeds 80 per device, and loads results into MySQL via Kafka and Apache Beam.

Master Kafka installation and basic operations, including configuring zookeeper and server, creating topics, and sending and consuming messages, setting up for Apache Beam integration.

Learn to consume Kafka messages with Apache Beam in real-time streaming mode by converting JSON IoT event data into a Java POJO, deserializing Kafka records, and printing device details.

Explore streaming with Apache Beam in Java by counting IoT records with temperature above 80 Fahrenheit using fixed 10-second windows, per device, after filtering, and sending results to Kafka.

Load streaming ETL data from Kafka into MySQL using Apache Beam and JDBC, implementing a fixed window per device count and writing the event name and count to the database.

Learn how to use Beam SQL in Java to query a data file by converting it to a PCollection, defining a schema, and applying select and aggregation operations.

Demonstrate using beam sql to count by user id with a group by, produce a csv output of user id and count, and introduce a subsequent join operation.

Learn to build a batch ETL pipeline on Google Cloud with Apache Beam in Java, using Storage and BigQuery, including account setup, bucket creation, data validation, transformation, and loading.

Sign up for a Google Cloud Platform free trial by creating an account with a Gmail, providing country, address, city, postal code, and payment details to start the free trial.

Sign in to Google Cloud Platform, create multi-region storage bucket with standard storage and Google managed key, then upload user.csv into input folder and generate JSON service account key.

Validate and transform big data by splitting rows, keeping records with seven columns, and filtering for India (ignore case) before loading into BigQuery in the next video.

Learn to ingest data into Google BigQuery with Apache Beam by creating a dataset and table, defining a schema, and inserting rows via a Beam pipeline.