Monitoring NiFi

Manoj G T | Big Data Evangelist
A free video tutorial from Manoj G T | Big Data Evangelist
Big Data Evangelist, JavaScript Lover & Web Developer
4.4 instructor rating • 1 course • 4,682 students

Learn more from the full course

Apache NiFi - A Complete Guide | Cloudera DataFlow | HDF/CDF

Apache NiFi is a robust Data Ingestion, Distribution framework & ETL Option. NiFi the core of Hortonworks Data Platform.

02:46:40 of on-demand video • Updated October 2020

  • Core Concepts of Apache NiFi to Design your Data Pipeline or ETL
  • Key NiFi Features like Back Pressure, Prioritization, Expiration, Reporting Task, Data Provenance, etc.
  • More Advanced Features of NiFi Like Cluster Setup, Monitoring, NiFi Registry, Custom Processors, etc.
  • Install Apache NiFi in Standalone Mode & Cluster Mode
  • Hands-on Experience with Apache NiFi
  • Using NiFi Registry for Version Control
  • Creating Custom Processors in NiFi
  • Monitoring in NiFi
English [Auto] Hi guys. Welcome back. In this video we will see how to monitor your knife for instance using the various options available in knife fight knife I use a powerful tool and it can do multiple tasks in battle. So it's essential to have a proper weight to monitor a knife for instance. This is where a batch in a face rich set off monitoring capabilities comes in handy. Let's go ahead and see these capabilities one by one. The first and foremost statistics are available in desktop status but often a few way yo we can see a few white those statistics about the current health off notify. For example the active threat stats shows the number of active threats currently running across the for instance. And the big hole heart knife fight is working for us the total your data specifies how many floor fights are present. Like you across the entire floor. And it also represents that total size of those floor files Yeah. We can also have stats of the total number of running processors stock processors and processes which are invalid due to some configuration issues if the knife for instance is in a cluster mode or a disconnected door in a faded history forward motion control. We will see more stack of sticks like how many node side in that Lester and o Many components are up to date are out of sync with the component version available in the industry. We will see more about clustering and my favorite history in the later videos where you will get a better understanding of what I am talking urinal knife I also provide stacks at individual component level each processor and process group on the canvas. It much more info about how much data has been processed by the component in the past few minutes. It also allows us to see the number of floor face that has been consumed by a processor as well as the number of floor face that has been produced by the processor in the last five minutes by default. Now if I ever take a snapshot of these five minute statistics every minute for 20 for us and keep it in a separate repository called US data supposedly you can view these detailed statistics by right clicking the component and just the view straight as a string option please be know that the default configuration of the status repository snapshot can be changed by updating the following properties in nice fade out properties file in addition to the statistics provided by each component knife I will also notify when any issues of type warning or error occurred in overflow. This is called bulletin. It looks like a sticky note and it will show all the issues produced by a processor in the past five minutes to demonstrate this. I created a simple flow with get file processor and configured it to point to an invalid folder. But now let's go ahead and start the flow. Did you see that we got an error here which is notified by the Bulletin indicator. Let's most over the bulletin to see the details of the error order warning message bulletin message comes handy for the user and saves a lot of time which usually takes filtering through a long fight to find errors in plus judgment. The Bulletin will also indicate which note into question and we take that. Edit Even you can change the log level of the bulletins by configuring the bulletin level of the processor configuration available in this setting step. We can also use the bulletin board option available in desktop write global menu to see all the bulletins that has occurred so far in the night for instance yo. We can also filter the bulletin based on the component name message and so on. That's it for this video guys. Hope you got a fair understanding on how to monitor your knife for instance using the various options available in a fight. Thank you. See in the next video.