Spring Batch Mastery

Basic Spring Batch Concepts: Job, Step, Reader, Processor, Writer and Job Repository

Temperature sensors business problem that our application is going to solve

Temperature sensors calculation job high-level design

Environment setup and configuring job repository db connection properties

Configuring the application with CommandLineJobRunner

Usage of the DefaultBatchConfiguration class in the setup of command-line based Spring application

Reader and writer responsibilities in Spring Batch. ItemReader and ItemWriter interface definitions

Diving into various ways of using FlatFileItemReader to read text files of literally any format

Instrumentation of the reader that reads temperature sensor data from the text file, leveraging LineMapper implementation supplied to the FlatFileItemReader

Definition of StaxEventItemReader & StaxEventItemWriter abstractions for reading data from XML format and writing data to XML format, respectively. Implementation of XML read & write capabilities of the temperature sensor job using StaxEventItemReader & StaxEventItemWriter with XStream library

Processor concept definition and ItemProcessor interface specification. Temperature sensor job specific implementations of the ItemProcessor, including filtering behavior for the second step.

Diving into various ways of using FlatFileItemWriter to write data in plain text files of literally any format, and leveraging FlatFileItemWriter to implement writing of the temperature sensor anomalies data to text file

Demo of the temperature sensor job application in both successful and failure scenarios

Overview of the chunk-oriented processing, and introduction of chunkSize (same as commitInterval) setting

Demonstration of temperature sensor job behavior based on various chunkSize setting values in failure / exception scenario

Introduction and purpose of Spring Batch entities: JobParameters, JobInstance, JobExecution and StepExecution

Comprehensive introduction of the ExecutionContext entity / abstraction and detailed analysis of how ExecutionContext works with jobs and steps in different scenarios, including failed jobs / steps and potential restore of the ExecutionContext on subsequent runs

Comprehensive and meticulous explanation of Spring Batch behavior in scenarios of repeated job starts using the same JobParameters, including the topic of job specific ExecutionContext re-use, and also covering preventRestart() setting effect for jobs in various circumstances

Detailed study of Spring Batch behavior in case of repetitive step (re)starts, in terms of whether to execute / skip execution, and whether to reuse / create new step specific ExecutionContext

The concept of skipping already processed items on restart, and controlling this behavior with 'saveState' setting

Avoid skipping previously completed steps with 'allowStartIfComplete' property, and how it affects the framework behavior in various circumstances

Capping the number of times specific step can be started with 'startLimit' property

Business problem of bank transactions that our Spring Batch application is going to solve

Technical design of the Spring Batch application solving bank transactions business use cases

Environment setup and configuring application properties to connect to job repository, input source database and unit / end-to-end test database

Examining the utility Gradle task to (re-)create input source data for the application

JobExecutionListener and StepExecutionListener interfaces, and their annotation analogues - @BeforeJob, @AfterJob, @BeforeStep and @AfterStep, as well as practical usage of beforeStep lifecycle capability to add new column to the table

Crash course on what are SQL cursors in relational databases

Inner details of JdbcCursorItemReader implementation, and implementation of reading from bank transactions table using SQL cursors approach with JdbcCursorItemReader

Concept of pagination approach for reading data from the database, along with technical details of how JdbcPagingItemReader works under the hood, plus usage of the JdbcPagingItemReader for reading data in 2 reporting steps of the application

Summary of database read approaches in Spring Batch, and high level overview of StoredProcedureItemReader, JpaCursorItemReader and JpaPagingItemReader

Stateless database reading design pattern discussed in details with visual examples

Implementation of currency adjustment feature of the application using the stateless database reading design pattern

The architecture of the driving query design pattern examined with visual examples

Overview of the Spring Batch capabilities to write data to the database. JdbcBatchItemWriter technical design and how-to-use (API). Implementation of:

• currency adjustment database write

• bank transaction balance calculation write

both using JdbcBatchItemWriter, with one using stateless database reading design pattern, and another leveraging out-of-the box approach

Technical design of using ExecutionContext as a shared variable container for filling the bank account balance, as well as details of the code implementing such approach

BatchStatus, ExitStatus, and its usage in building conditional flows in Spring Batch. Implementation of the conditional flow in the bank transaction analysis application leveraging StepExecutionListener

Spring Batch capabilities for reading & writing JSON file format, as well as code implementation leveraging JsonFileItemReader and JsonFileItemWriter for aggregation report generation feature of the application

Demonstration of the run / execution of the bank transaction analysis application developed during this section

Various Spring Batch capabilities to implement step execution early exit, including ItemReadListener (code demonstrated) and .setTerminateOnly()

In-depth analysis of capabilities and tools of the Spring Batch to test steps and jobs end-to-end, demonstrated with JUnit end-to-end tests covering the code of the bank transaction analysis application

Code walkthrough of unit test capabilities of Spring Batch, covering mocking technique implemented both with MetaDataInstanceFactory out-of-the box approach and with Mockito library

JobLauncher concept, corresponding JobLauncher interface API, and walkthrough of commonly used implementation, TaskExecutorJobLauncher

Teams and players sports performance business problem that our Spring Batch application is going to solve, with specific product and technical requirements set

Technical design of the Spring Batch Web Application that is going to solve team and player performance business problem

Environment setup and configuring application properties to connect to job repository

Practical explanation (and code walkthrough) on structuring the application as web app using Spring Boot, with endpoint definitions, asynchronous job start setup using JobLauncher, and further ways of integrating the application with different technologies to trigger job executions

Technical design and implementation of the custom multi-line ItemReader (text file format), and using that in the application in conjunction with MultiResourceItemReader

ItemStream interface and its role in Spring Batch architecture, as well as its interplay with custom multi-line ItemReader (text file format) and MultiResourceItemReader in the team and player performance application

Manual on using MultiResourceItemReader to read from multiple files / directory with one reader

Uncovering details of accessing JobParameters values with @Value and @StepScope annotations, and using that in the ItemProcessor implementation to calculate team average scores

The layout and technical concept of execution context property promotion design pattern; ExecutionContextPromotionListener class and its usage to promote min / max calculated scores and names; average score calculation step ItemWriter and logger callback implementation(s)

Accessing job specific ExecutionContext properties with @Value and @StepScope annotations, and using these properties to implement ItemProcessor for min and max ratio parallel steps of the application

Header and footer concepts for flat files, FlatFileHeaderCallback & FlatFileFooterCallback interfaces provided to write headers and footers, and implementing min and max step writers using FlatFileHeaderCallback to populate the information in header of the file

Introduction of the special tasklet steps with no reader and writer, as well as corresponding Tasklet interface and its usage in the application code to execute system command (using CommandRunner interface and JvmCommandRunner implementation) and log informational message

Details of building graphs of steps with Spring Batch using key building block interface Flow, as well as the implementation of split parallel flow of min and max ratio steps using ThreadPoolTaskExecutor

High level overview of Spring Batch retry capabilities, including faultTolerant(), retry(), noRetry() and retryLimit() settings, as well as RetryPolicy interface

Demonstration of the team performance application run both is successful and failure scenarios, with taking a thorough look on failure scenario with using skip capabilities of the Spring Batch, including faultTolerant(), skip(), skipLimit() and noSkip() settings, as well as SkipLimitExceededException scenario

Walkthrough over step lifecycle listeners that were not covered before: SkipListener (including application code implementation that logs skipped items), ItemReadListener, ItemProcessListener, ItemWriteListener and ChunkListener (and comparison of 2 latter)

Analyzing and implementing 2 ways of testing @StepScope defined components:

1. Explicit StepExecution resulted method defined

2. Using SpringBatch provided StepScopeTestUtils

both covering teams and players performance application code with JUnit

Introducing JobExplorer concept and interface, and comparing it to JobRepository

Action calculation business problem that our application is going to solve, as well as its high level technical design

Multiple aspects covered:

• environment setup

• configuring application properties to connect to job repository and source database

• working with input data generator task

• skeleton of the web app which uses Spring Boot

• controller and endpoint definitions

Technical design of the job running in a single thread solving action calculation problem, including the trick with Postgres upsert SQL query

Single thread job application code thorough explanation, end-to-end, followed by demonstration of the code executed to calculate user action scores

Multi-threaded job technical design focusing in depth on thread safety and synchronization aspects of readers and writers

Multi-threaded job application code thorough explanation, including introducing SynchronizedItemStreamReader and SynchronizedItemStreamWriter. Demonstration of the code working, followed by detailed analysis of 2 problems:

• Deadlock handling

• Exact ordering requirement

The concept and technical design of remote chunking design pattern used by the Spring Batch framework explained

Opinionated analysis of pros and cons of Remote Chunking Spring Batch approach, with focusing on overhead, questionable responsibilities separation and tight coupling with Spring Integration framework

The concept of partitioning discussed thoroughly keeping in mind action calculation problem, as well as PartitionStep entity is introduced

Detailed explanation of the essential core Spring Batch provided interfaces, classes and their respective roles aimed to implement partitioned jobs on top of, including PartitionStep, PartitionHandler, StepExecutionSplitter, StepExecutionAggregator and DefaultStepExecutionAggregator

Deep dive into an SDK classes and interfaces of Spring Batch, clarifying their roles and responsibilities in both generic and local multi-threaded execution cases, e.g. AbstractPartitionHandler, TaskExecutorPartitionHandler, SimpleStepExecutionSplitter and Partitioner

Covering the application code for partitioned local job in details, including custom Partitioner implementation, as well as demonstration of the execution with reviewing the application logs

Solving the way Spring Batch stores ExecutionContext for partition workers through connecting to and querying JobRepository database

Detailed technical architecture and design of the distributed partitioned Spring Batch job for the action calculation problem

Implementing the endpoint to start worker partition step asynchronously (as a part of larger partition job)

Unfolding the technical design of the distributed job discussed above into interfaces and classes that will be used, comparing that to partitioned local job domain design

Thorough exploration of the distributed job application code, including custom implementation of the PartitionHandler interface, followed by demo of the 3 instances accomplishing the partitioned distributed job successfully