An Advanced Guide for Apache Hive: A Hadoop Ecosystem Tool

Explore big data as volumes of structured and unstructured data that exceed traditional processing, arrive from sources, vary in formats, move with velocity, and require quality for credible results.

Hadoop enables distributed processing of large data sets across clusters using simple programming models, scalable from a single server to thousands with Hadoop distributed file system and map and reduce.

Explore how HDFS distributes data across a Hadoop cluster, leveraging block replication, fault tolerance, and a master–slave architecture with name nodes and data nodes.

Explore how Hadoop MapReduce processes large data sets on commodity clusters using Java. Map inputs into key-value pairs and reduce them to final results, illustrated by a word count example.

Explore how Hive handles Twitter-scale data, showing why traditional RDBMS fails, and how Hive tables can be partitioned and bucketed with schema evolution and ODBC/HD API support.

Contrast Hive and MapReduce within the Hadoop ecosystem, showing Hive's shorter code, faster development time, easy joins, and command-line based job submission with no compiling or job files.

Explore Hive, an Apache data warehouse tool in the Hadoop ecosystem that enables storing and querying large datasets with a SQL-like language. Enable user-defined functions and leverage execution plans.

This lecture describes HiveQL advantages, showing SQL-like querying, support for custom UDFs, diverse data formats, and access to HDFS and HBase for batch processing.

Explore the Hive architecture, from the interactive shell and driver to the compiler, metastore, and execution engine, which translate queries into a directed acyclic graph of map-reduce tasks.

Explore Hive data types, including primitive types such as boolean, string, timestamp, binary, and null, and complex types like maps, structs, and unions, with string quoting and null handling.

Explore Hive query language and data definition language in Apache Hive, using create, alter, and drop commands to manage databases, tables, views, functions, and indexes.

Explore DDL commands in Hive to create, use, describe, alter, and drop databases, set and view properties and extended properties, and understand cascade behavior and the default database.

Learn to create hive tables with create table if not exists, define a schema (name, salary, address), copy schemas with like, and describe formatted.

Explore Hive's managed and external tables, including partitioned tables and external partition tables, and learn to create them with location, schema, and partitioning rules.

Explore advanced ddl on tables in Apache Hive, including drop table if exists, rename, add or drop partitions on partitioned tables, modify or add columns, and set properties.

Explore file formats in Hive and how data is stored, with text, sequence file, and RCFile options; specify storage format when creating tables, such as employee stored as text.

Load data into Hive tables using DML commands, from local files or HDFS, then insert into existing or partitioned tables by specifying partition values.

Explore managing outputs in Apache Hive by inserting results into new tables and using static and dynamic partitions, and exporting data to local directories in a single statement.

Explore HiveQL queries, distinguishing select data retrieval from actions like insert or delete. Project fields with select from, apply aliases, and access map values like deductions.

Explore Hive operators and functions, including arithmetic, predicate, and bitwise operators, along with string and aggregate functions like round, floor, max, min, abs, and substr, in practical hands-on examples.

Master hive clauses: limit, where, like, with column aliases and nested selects. Apply calculations, case when, group by, having, and order by to refine results.

Explore how Apache Hive handles joins, including inner, left outer, right outer, full outer, semi, cross, and map joins, with practical examples using customers and orders.

Create an ms-word database and a table in hive, load text as a string, split into words with explode, and count occurrences by word.

Learn how to create and manage Hive views, which are read-only logical subsets of table data defined by as select, without storing data, supporting if not exists and properties.

Create unique Hive views within a database, exposing only selected columns like first name, last name, and address to protect passwords, using where clauses and maps for dynamic views.

Explore advanced Hive clauses, including distribution by and cluster by for sorting, casting for data type changes, and union all to combine tables.

Explore how Hive indexing speeds queries by referencing specific columns, with compact and bitmap indexes stored separately and can be created, rebuilt, or dropped on tables like employee.

Learn how to create indexes with additional properties on the employee table, specify index properties, and explore bitmap indexes in Apache Hive.

Tune hive performance by using explain and explain extended to reveal execution plans, optimize joins with map joins and stream table optimization, and apply practical tuning knobs.

Explore Hive execution modes to balance scalability and overhead, enable local mode for runs, and use strict mode to avoid full table scans with where, order by, and limit.

Learn how to tune Hive performance in a Hadoop ecosystem by enabling JVM reuse, configuring dynamic partitioning, and enabling speculative execution to accelerate tasks and handle many short tasks.

Explore how compression with Hive saves storage and speeds data transfer by using various compression schemes, while Hive reads and writes multiple formats without forcing a specific format.

Explore compression codecs in Hive, including the default snappy and deflate, plus gzip and bzip2. Weigh speed against compression ratio to select the right codec for I/O and disk space.

Explains sequence files in Apache Hive, their block-based compression options, benefits for space and split processing, and the header fields like version, key/value classes, and sync markers.

Explore sequence files and their compression types: none, uncompressed, block-based, and record compression. Learn how header, records with keys and values, and sync markers organize data, including block-based formats.

Learn to enable intermediate and final compression in Hive, configure compression codecs, and verify compression by querying tables and inspecting the compressed output behind the scenes.

Explore how Apache Hive functions work, including built-in and user defined functions implemented in Java, loaded into your session, and described with describe and describe extended.

Explore the different kinds of functions in Hive, including standard scalar functions, user-defined functions, aggregate functions, and user-defined table generating functions, with demonstrations of their usage in queries.

Explore Hive's user defined functions: scalar UDFs, user defined aggregation functions, and user defined tabular functions, demonstrating when to plug in your own logic for complex results.

Create a Java-based user defined function for Hive, export it as a jar, and use it like built-in functions to remove extra spaces and escape sequences from strings.

Explore Hive file formats, including default text files and sequence files, and how stored as text or sequence shapes input, output, and sharing with Hadoop tools.

Explore RCFile format in Hive, a columnar storage that compresses low-cardinality columns for efficient storage and fast query processing; learn to create RCFile tables, load data, and query columns.

Learn csv and json serdes in Hive to convert unstructured data into hive-ready records, configure the jar and fully qualified class name, and map fields to columns.

Explore Hive Avro SerDe, a schema-driven serialization system that enforces Avro schemas on read, supports nested schemas, and translates into Hive types while handling compressed files for backward-compatible queries.

Understand how storage handlers in Hive enable seamless data access from diverse sources through input and output formats, supporting systems like text files, Cassandra, and Dynamo.

Explore Apache HBase, a distributed, scalable database built on Apache for storing data in massive tables and enabling fast random reads and writes.

Learn to load data into hbase via apache hive by creating a hive database and tables, importing data with the DSV method, and mapping rows to column families and qualifiers.