Spring Cloud Data Flow - Cloud Native Data Stream Processing

• Introduction to Spring Cloud Data Flow dashboard

• Features of Dashboard

• Working with Apps, Runtime, Streams, Tasks and Jobs

• Creating sample stream using time source and log sink

• Deploying the stream and configuring the properties

• Verifying the message broker, sink log files

• Registering application using Task app starters

• Creating task with Timestamp app

• Configuring and deploying the tasks

• Verifying the execution of tasks with start time, end time and exit code

• Verifying the log files for result

• Creating AWS EC2 instance with CentOS operating system

• Login to EC2 instance using Putty

• Install required packages and updates

• Installing Docker service and starting

• Installing Docker compose

• Downloading required docker compose yml file

• Overview of various service in Spring Cloud Data Flow docker yml file

• Customizing yml file

• Setting required environment variables

• Bring up docker compose

• Verifying list of running services

• Create and verify sample stream using timestamp and log

• Verifying Grafana dashboard for results

• Check status of various container

• Login to skipper server

• Check log files of streaming application in skipper server

• Login to spring cloud data flow server and data flow shell

• Login to mysql container and check the database created by Spring Cloud Data Flow Server

• Terminate the instance

• Adding Spring Cloud Stream App Starters

• Create and populate database data

• Create stream with datasource and driver configuration

• Post sample data with http post method

• Check the logs and database for verification

• Deploy stream using webUI

• Post data using postman application

• Verify data in Grafana dashboard

• High level overview of twitter stream application layout

• Set up Twitter Authentication details Consumer API keys and tokens, Access token and access token secret

• Create twitter stream with configured authentication details

• Deploy the stream and check the logs for harvested tweets

• Verify structure of harvested JSON structure

• Validate twitter JSON for language expression using JSONPath Syntax

• Extract language value and count using counter processor

• Use fan-out architecture to get the language counter along with twitter logs

• Create new panel in Grafana dashboard

• Edit Query to have pie chart on languages count

• Try various types of graphs like Bubble chart

• Monitor real time data getting reflected in dashboard

Creating JSONPath expression to extract hash tags

Creating stream to count hash tags using counter processor

Importing template to Grafana

Visualizing the twitter harvested information in Grafana dashboard

• Designing stream to extract twitter for sentiment analysis

• Overview on using pre-built model for sentiment analysis

• Configuring Vocabulary and model for sentiment analysis

• Deploying sentiment analysis stream

• Verify real time tweet sentiment in Grafana dashboard

• Overview on Object Detection Tensor Flow porcessor

• Registering starter application

• Creating and deploying stream using data flow shell

• Adding source image and verifying detected object in result image

• Trying various samples like animals and birds

• Overview on Pose estimation Tensor Flow Processor

• Creating and deploying stream for Tensor Flow pose estimation processor

• Verifying pose estimation with sample image

• Overview on Spring Cloud Task

• Creating starter project using spring Initializr

• Importing project into Spring Tool Suite (STS)

• Creating class implementing ApplicationRunner interface

• Creating required pom dependencies

• Verifying the application in STS

• Building jar file using maven build

• Hosting the application in web server and registering it as app in spring cloud data flow server

• Verifying the result in log file

• Overview of processor application architecture

• Role played by binder and messaging middle ware

• Creating bootstrap application using Spring Initializr by adding required dependencies

• Importing the project into Spring Tool Suite (STS)

• Using the annotations like @EnableBinding, @Transformer

• Verifying the queues getting created in RabbitMQ

• Verifying by posting sample message in RabbitMQ

• Build custom processor as jar file and register in maven local repository

• Deploying application using maven uri maven://

• Registering custom created processor

• Creating and deploying stream using http source, custom processor and log sink

• Verifying deployed stream using http post method

• Creating the stream using Shell and Web UI

• Architecture and working of Source application

• Role played by poller

• Creating Initializr application and importing to STS

• Using annotations like @EnableBinding, @InboundChannelAdapter, @Poller

• Creating sample message with payload

• Building and registering the application in local repository

• Registering custom source application

• Deploying stream and verifying result in log file

• Creating application in Spring Initializr and importing into STS

• Using annotations like @EnableBinding, @StreamListener

• Building and registering jar with local repository

• Verifying the result in log file with http post

• Creating custom processor

• Creating stream with custom source, processor and sink

• Stream architecture and flow details

• Registering all three (Source, Processor and Sink) application with spring cloud data flow server

• Verifying the stream with sample data