Bigdata with AWS - From Beginner to Expert

Explore how Hadoop enables distributed storage and processing of big data across commodity clusters, using MapReduce and the Google File System concepts, with 1.x architecture featuring name and data nodes.

Explore Hadoop 2.x architecture, detailing active and standby name nodes, data node heartbeats, and block writes, with Zookeeper, fencing, and checkpoint mechanisms that prevent split-brain and ensure fault tolerance.

Learn core hdfs commands in a Cloudera cluster: differentiate dfs from local fs, create directories, copy files between local and hdfs, and read or append data.

Learn HDFS commands to manage data, listing and opening files, copying and renaming, and configuring permissions, replication factor, and block size in a hands-on workflow.

explore hdfs commands for checking file existence, understanding exit status, computing checksums, adjusting replication factor with setrep, and transferring data with get and put.

Sqoop introduces a wrapper for MapReduce that moves data between relational databases and Hadoop. It supports import and export to HDFS, Hive, and HBase, with incremental and parallel loading.

Master Sqoop basics by connecting to a local MySQL database, listing databases, selecting data, and performing import and export with boundary-based splitting and basic syntax guidance.

this part introduces sqoop in aws, showing how to split workload across mappers using primary keys and min/max boundaries, and how to handle passwords securely during import and export.

This lecture demonstrates Sqoop import techniques, including direct import, target directories, and append operations, with selective columns and all-tables imports across MySQL, PostgreSQL, and Oracle.

Explore Sqoop: import data from a database into Hadoop, manage primary keys, exclude records, choose binary or sequence formats, and apply incremental load options.

Execute incremental imports with Sqoop, configure import jobs, and automate data updates using timestamps to manage records in the database and generate reports.

Explore part 6 Sqoop, focusing on implementing incremental imports, creating and updating jobs, and using a check column and timestamp-based logic to capture new and modified records.

Explore Sqoop export workflows from relational databases to external storage, including incremental imports, data type mapping, and handling nulls to ensure clean serialized data.

Explore Sqoop export workflows, focusing on update only mode and the allow insert option, using staging tables to export and update data between file system and relational databases.

Discover Hive, a data warehouse ecosystem on Hadoop that enables querying large data with SQL-like Hive query language, converting queries into map reduce jobs for analysis and summarization.

Explore Hive basics by creating a database and a table with properties, then load data from local and remote sources, examining serialization and metadata.

Explore Hive data types like arrays, maps, and structs, and learn to query and access collection elements and map keys using indexing in Hive tables.

Explore Hive part 4 concepts: managed vs external tables, location usage, and how data is loaded or pointed to without loading, with the Hive warehouse in mind.

Explore Hive time and string functions in big data with AWS, including unix_timestamp conversions, date formatting, date differences, and common operations like split, trim, and substr.

Delves into Hive analytic functions and window ranking, and explains static and dynamic partitioning in Hive, showing how partitioning speeds queries by scanning only relevant data.

Explore Hive partitions, focusing on static partition setup, partition by country, and loading data through intermediate tables, with tips on dynamic partition handling and querying partitioned data.

Explore dynamic versus static partitions in Hive, learn how to enable dynamic partitioning with required properties, and apply them to partitioned and bucketed data loading scenarios.

Master Hive joins by loading three tables, joining customers with orders using normal, left, and right joins, and determining how many order items each customer placed.

Explain Hive join types, including map join and bucket map join, and how partitioning, bucket alignment, and build-side data size affect performance in practical join scenarios.

Master Hive transactions and ACID semantics in big data workflows. Learn enabling transactional tables, ORC storage, bucketing, and controlled insert, update, and delete operations with the Hive transaction manager.

Explore how big data storage and schema design intersect with relational concepts, using functions, offset, and sorting to retrieve salary and department data in a Hive context.

Explore how Hive supports data management through metadata, RDBMS integration, and practical command-based workflows. Apply concept-driven approaches to design and troubleshoot data processes.

Introduce serialization and deserialization, compare row-based and column-based file formats, and cover Avro, Parquet, and ORC, including schema evolution, compression, and Hive suitability.

Explore Hive basics through practical examples of jobs and JSON objects, and compare two approaches for creating tables and querying data in Hive.

Scala blends object-oriented and functional programming to form a scalable, JVM-based language by Martin Aldersgate, featuring static typing with type inference and Java interoperability.

Explore scala data types, type inference, immutable vs mutable, and java interoperability via any and anyref. Learn basic data structures and string interpolation patterns.

Learn Scala control structures and object oriented basics in this Part 3 lecture, covering while and do-while loops, for loops, infinite loops, and companion, anonymous, and single objects.

Master scala's case class to model immutable data, map raw objects to a schema with map, and leverage pattern matching for scalable data handling in distributed dataset.

Master interview-ready Scala concepts, including closures, higher-order functions, traits, and pattern matching with case classes. Explore collections and exception handling, plus Spark basics for data engineering applications.

Explore Apache Spark, a blazing in-memory computing engine that overcomes MapReduce limits with lazy evaluation and real-time streaming, featuring Spark Core, Spark SQL, Spark Streaming, MLlib, and GraphX.

Explore spark architecture, including the driver program, cluster manager, executors, and containers, and see how DAGs convert code into jobs, stages, and tasks.

Explore Spark fundamentals from Spark context and resilient distributed datasets (RDDs) to transformations and actions, including map, flatMap, filter, groupByKey, reduceByKey, and repartition.

Master Spark on AWS by spinning up a cluster, creating notebooks, and loading data. Build dataframes and perform joins on orders and order items to extract insights.

Master spark data frames built on RDDs, understand their immutable nature, create data frames from raw data with case classes or explicit schemas, then run SQL and DSL queries.

Explore Spark dataframes with the column function to rename, change, or derive columns. Practice exploding arrays, handling nested structures and maps, and performing group by, aggregations, and order by.

Explore Spark SQL joins, including inner, left, right, full, plus self and semi joins, unions, and window functions with partitioning to drive scalable data processing in Spark.

Explore how HBase, an open-source, column-oriented, distributed database built on HDFS, stores data in column families and delivers near real-time read/write with region servers, masters, and zookeeper.

Learn how to integrate spark with hbase by reading from hbase and writing back, using a three-step process to define schema, catalog, and column mappings.

Learn Spark streaming with DStreams by building a Twitter data pipeline using a streaming context, and applying tumbling and sliding windows to count top hashtags in real time.

Explore Spark structured streaming concepts, define schemas, read streaming data from folders, and output results to the console with various modes and triggers.

Kafka is a fast, scalable, fault-tolerant distributed messaging system that implements publish-subscribe with topics and partitions, managed by brokers and ZooKeeper, enabling producers, consumers, offsets, and consumer groups.