Apache NiFi Complete Master Course - HDP - Automation ETL

• Create a simple Flow

• Introduction to GetFile and PutFile Processor

• Processor Configuration

• Connection Configuration

• Relationship Termination

Understand various Category types like

• Data Ingestion

• Routing and Mediation

• Database Access

• Attribute Extraction

• System Interaction

• Data Transformation

• Sending Data

• HTTP Access

• AWS Cloud Access

Understand various configuration option as part of connections like

• Flowfile Expiration

• Back Pressure

• Object Threshold

• Size Threshold

• Prioritization

• Various options in connection context menu

• Queue monitoring

• Using Queue Empty Options

Various general configuration as part of Processor Settings tab

• Penalty Duration

• Yield Duration

• Bulletin Level

• Relationship Termination

Various options in Processor Scheduling Tab option

• Different Scheduling Strategy

• Relationship between latency and throughput

• Concurrent task configuration

• Run Schedule configuration

• Execution mode

• Managing property of various processors

• Customizing mandatory and non mandatory properties

• Error handling on missing properties

• Changing the payload attributes

• Taking decision based on flowfile attribute

• Logging the attributes in log file

• Monitoring log attributes

• Customizing log file configuration

• Logging attributes in separate log file

• Failure handling by processor

• Retry failed Flowfile

• Monitoring failure queue

• Check failure message from bulletin

• Purpose and use of Templates

• Creating Templates

• Managing Templates

• Uploading Templates

• Template file structure

• Handling sensitive information in Templates

Understand Apache Kafka

Install Kafka

Create Topic

Publish Message to Topic

Read Message from Topic

• Create message with Flowfile

• Post message to Kafka Topic

• Read message using Kafka console consumer

• Post message using Kafka Producer

• Read message from Topic using Apache Nifi

• Convert message to Flowfile

• Purpose of process group

• Input and Output Ports

• Create and Use Process groups

• Create Funnel

• Understand forking concepts and its use

• Fork flow file to multiple processor

  
  

• Understand Combine or Fan-in concept and its use

• Combine flowfiles from multiple processors

Monitor various statistical information about

• Processors

• Input Ports

• Output Ports

• Remote Process Group

• Connections

• Process Groups

Observing overall Bulletin

Data Provenance

Nifi history

• Purpose and Usage of Data Provenance

• Provenance data lineage

• Detail event analysis

• View/Download input and output claim

• Replay / Retry events

• Observe failed queues

• Observe modified attributes as part of event

• Connect and Read data from MySQL database

• Use of Avro and JSON

• Using Connection Pool Controller

• Using AvroSchemaRegistry

• Using CSV Reader

• Using JSONRecordWriter

• Using Custom Schema of CSV

• Monitor updated attribute using data provenance

• Purpose of state management

• Reading only Delta records from RDBMS table

• Using maximum-value Column property to manage state

• Creating sample data from mackaroo

• Using dynamic schema

• Purpose and realtime use of dynamic schema

• Nifi - Registry Introduction

• Purpose of Nifi Registry

• Installing Nifi Registry

• Staring Nifi Registry service

• Creating and managing buckets

• Connecting Nifi Registry with flow file

• Adding a flowfile to a bucket

• Maintain Version of flowfile

• Committing and changing version of flowfile

• Check version history of flowfile

• Rollback changes

• Install 3 node cluster

• Understand Primary Node, Cluster Coordinator role and responsibility

• Using Zookeeper Quorum and configuration

• Configuration change of nifi.properties and state-management.xml

• Update Zookeeper connection string

• Starting nodes and cluster

• Verification of primary node election

• Cluster overview on Nodes, Systems, JVM, Storage and Versions

• Create sample flow file

• Monitor status history

• Execution of processor in Primary node and in All nodes