Bigdata with AWS - From Beginner to Expert

Learn core hdfs commands in a Cloudera cluster: differentiate dfs from local fs, create directories, copy files between local and hdfs, and read or append data.

Learn HDFS commands to manage data, listing and opening files, copying and renaming, and configuring permissions, replication factor, and block size in a hands-on workflow.

explore hdfs commands for checking file existence, understanding exit status, computing checksums, adjusting replication factor with setrep, and transferring data with get and put.

Master Sqoop basics by connecting to a local MySQL database, listing databases, selecting data, and performing import and export with boundary-based splitting and basic syntax guidance.

Master Hive joins by loading three tables, joining customers with orders using normal, left, and right joins, and determining how many order items each customer placed.

Explain Hive join types, including map join and bucket map join, and how partitioning, bucket alignment, and build-side data size affect performance in practical join scenarios.

Explore how Hive supports data management through metadata, RDBMS integration, and practical command-based workflows. Apply concept-driven approaches to design and troubleshoot data processes.

Explore Apache Spark, a blazing in-memory computing engine that overcomes MapReduce limits with lazy evaluation and real-time streaming, featuring Spark Core, Spark SQL, Spark Streaming, MLlib, and GraphX.

Explore Spark fundamentals from Spark context and resilient distributed datasets (RDDs) to transformations and actions, including map, flatMap, filter, groupByKey, reduceByKey, and repartition.

Explore Spark structured streaming concepts, define schemas, read streaming data from folders, and output results to the console with various modes and triggers.