Apache NiFi (Cloudera DataFlow) with Free Lab

Install NiFi by downloading, extracting, installing Java, and starting services; then retrieve credentials from nifi-app.log and log in to web user interface at https://server:8443 with port changes via config file.

Navigate the NiFi user interface, featuring the component toolbar with processors, process groups, templates, and labels, plus the status bar, canvas, search bar, and breadcrumbs.

Discover Apache NiFi common terminologies, including dataflow and the dataflow manager, and learn how processors connect through relationships like success and failure to form data flows.

Master edl in Apache NiFi by reading files from a source, applying transformation, and loading them to a destination, using a simple input to output workflow.

Discover how to define an Avro schema for UniFi data by using a record type, a namespace and name for uniqueness, and detailing fields with name and type.

Learn to perform a record-level transformation in Apache NiFi by reading a CSV, applying an uppercase first-name change with UpdateRecord, and writing the result to a file.

Learn to enrich csv data with NiFi by reading employee.csv, performing country lookups via Avro schema registry, and partitioning outputs by country with unmatched handling.

Create a new full name column with the update record processor by concatenating first name and last name using the concat function on record values.

Explore NiFi functions using official documents, learn to apply expression language, and create unique file names with timestamps via update attributes.

Learn to partition data by gender using the partition record processor, creating separate files for male and female employees, with configurable record reader and writer and practical handling of delimiters.

Explore NiFi's Split Record processor to split CSV records into fixed counts, reuse controllers for CSV reader and writer, update file attributes, and route outputs to a Splitted Records directory.

Demonstrate splitting a large file into smaller flow files with NiFi, using csv reader/writer and GenerateFile/MergeRecord, stepping from 10,000 to 1,000 to 100 to 10 records.

Identify and prevent file level duplication using cryptographic hash attribute and distributed map cache, routing non-duplicates to putfile while duplicates are logged.

Learn to fetch json data from the restcountries api using InvokeHTTP, extract country names and languages with EvaluateJsonPath, and flatten or split json for analysis.

Identify record level duplicates by hashing the content of each split record with cryptographic hash content using SHA-256 or SHA-512, then detect duplicates for routing.

Discover how Apache NiFi reads from and writes to Hbase using Put Hbase Record and Get Hbase, with csv reader and environmental-tax-account schema in a schema registry.

Learn to read and write data to HDFS with NiFi, configuring Hadoop via core-site.xml and hdfs-site.xml, and using GetFile, PutHDFS, and GetHDFS.

Learn to run your own custom script with the Execute Stream Command processor to process flowfiles via standard input and output for normalization of pipe-delimited data.

Learn how to speed up NiFi development by using the Query Record Processor to run SQL queries on flow files, selecting specific columns from CSV data with header-based schema.

Learn to receive data over HTTP with NiFi's Listen HTTP processor, merge multiple flow files using MergeContent (minimum entries = 5), and verify the merged output and Merge.Count attribute.

Learn to build a NiFi web service using HandleHttpRequest and HandleHttpResponse, fetch file content with FetchFile, and return it via http response, driven by curl requests.

Discover how NiFi uses the H2 database by default, access the flow audit database with the H2 GUI client, and enable remote connections by editing NiFi.properties.

Learn to read CSV data and write it to a PostgreSQL database and environmental tax account table with NiFi, using GetFile, PutDatabaseRecord, CSV reader, and DBCP.

Connect to Oracle using the NiFi Query Database Table processor and configure a DBCP connection pool and Oracle driver to read incremental data with a maximum value column.

Set up NiFi Registry, create buckets, register a Registry client, and manage versioning of NiFi flows with commits and imports from NiFi Registry.

Learn to modularize NiFi pipelines by turning process groups into templates, share via template manager and NiFi Registry, and route data between groups using input and output ports.

Configure a reusable data flow with a configuration file controlling countlines, checkduplicate, doarchive, and hdfsload. Demonstrate building, templating, and deploying with a decision tree and attribute updates.

Explore NiFi provenance data, provenance repository, and data lineage by pulling provenance with site-to-site reporting, using an input port, and viewing json-formatted audit details of flowfiles.

Learn how to use NiFi processors CountText, LogAttribute, LogMessage, AttributeToCSV, and AttributeToJson to log attributes, count text, and produce CSV or JSON logs for auditing and reporting.

Understand NiFi clustering by coordinating multi-node operation through a cluster coordinator with heartbeats to avoid a failure point. Compare cluster and standalone setups and see flows run on nodes.

Practice reading excel files in NiFi by using the convert excel to csv processor, selecting the sheet name, and wiring get file to produce a csv output.