Event-Driven Microservices: Spring Boot, Kafka and Elastic

Set up the development environment by installing Java 25, Maven, IntelliJ, Git, and Docker, verify installations with version commands, and configure path and Javahome for permanence.

Discover generative AI with Spring AI to create tweet content without Twitter API, and learn model selection, prompt engineering, and using OpenAI, Vertex AI, or local llms like Llama.

Explore generative AI and large language models (LLMs) and their role in content creation, from text and images to code, with steps like pre-training, fine-tuning, and RAC.

Learn how to choose the right LM model for a task by weighing requirements, budget, and accessibility, and compare open source versus commercial options for privacy and cost.

Explore prompt engineering techniques to optimize interactions with large language models, including zero-shot, few-shot, chain-of-thought, system messages, prompt templates, temperature and top-p controls, with iterative refinement and context-aware prompts.

Enable seamless ai integration in Spring Boot apps with Spring AI, offering auto configuration, prompt templates, and real-time streaming responses from OpenAI and local llms.

Discover how Spring Boot enables lightweight standalone apps with auto configurations and starter modules. Explore web, test, actuator, security, data JPA, Thymeleaf, and cloud starters such as OAuth2 resource server.

Configure logback with console and file appenders, pattern, and rolling policy to print AI-generated keywords from config for the AI tweet to Kafka service, while comparing injection methods.

Obtain and configure OpenAI, Google Vertex AI, and DeepSeek API keys, set environment variables, and integrate them into a Spring Boot app to generate TV content.

Develop a skeleton microservice that implements the command component of a CQRS and event-sourced architecture, streaming AI-generated tweets to Kafka using Spring Boot and multiple AI providers.

Learn to generate tweet-like data via direct HTTP calls to the OpenAI API in a Spring Boot microservice, configuring dependencies, prompts, and OpenAI request/response models for Kafka publishing.

Generate tweet content by building a keyword-driven prompt from configuration and sending it to the OpenAI API via direct HTTP requests in a Spring Boot microservices setup.

Learn to implement an AI service with the OpenAI Java library, switch between OpenAI service and OpenAI Java client via config, and generate tweets using chat completions.

Enable tweet generation with Google Vertex AI by integrating the Google AI dependency, configuring project, location, model, max tokens, and temperature, and implementing a Vertex AI backed AI service.

Generate tweet content using Spring AI and OpenAI within a Spring Boot microservices setup, configuring dependencies, the OpenAI client, and prompt templates to produce structured tweets.

Illustrates switching from OpenAI to the DeepSea API via Spring AI, updating the API key and base URL, and using the Leipzig chat model to generate tweet data.

Generate tweet data locally using spring ai with Olama and the Leipzig R110b model, connected to a local Olama instance for Kafka.

Stream tweets from Twitter to Kafka using a mock Twitter stream, with a Spring Boot Twitter Kafka stream runner as the command component in a CQRS and event-sourcing setup.

Learn to stream tweets using the Twitter v2 API in a Spring Boot microservice, configuring v2 endpoints, bearer tokens, and streaming rules, with a mock alternative for free streaming.

Learn to implement a mock Twitter stream with MockKafkaStreamRunner, switchable via Spring Boot conditions, and configure mock tweets to run on a dedicated thread using a configurable delay.

Explore Apache Kafka as an immutable event store for an event-driven microservices architecture, covering topics, partitions, producers and consumers, replication, throughput, and key producer configurations.

Create a reusable app-config-data module to hold configuration data classes, enable externalized configuration across microservices with Spring Boot, and adjust component scanning and dependencies for seamless integration.

Set up a local Kafka cluster with Docker Compose, including Zookeeper, Schema Registry, and three brokers for quorum; monitor with kafkacat and scaffold submodules kafka-model, kafka-admin, and kafka-producer.

Build a kafka-model java object for kafka communication using Avro schema types. Configure Avro dependency and avro-maven-plugin in pom.xml, define Avro schema, and generate the model via mvn clean install.

Develop a kafka admin module with the admin client to create and verify topics, and implement retry logic with Spring Retry and Spring AOP for startup readiness.

Develop and validate a Kafka admin module that creates topics programmatically, with retry logic, topic status checks, and schema registry health checks, while integrating a web client for REST calls.

Implement a spring kafka producer module using confluent kafka avro serializer, add dependencies and exclusions, and configure producer properties in application.yml with key/value serializers and acks all.

Learn to implement a Kafka producer module by configuring a Kafka template and producer factory, wiring serializers and Avro schema, and handling asynchronous send callbacks for reliable event production.

Integrate the Twitter microservice with Kafka by using admin and producer modules to create topics, verify schema registry, and transform Twitter status to Avro for sending to Kafka.

Create a docker image for the twitter-to-kafka microservice using Spring Boot build-image, then run the dockerized Kafka cluster and services with compose and validate with kafkacat.

Implement externalized configuration for microservices with spring cloud config server by creating a config server repository and initializing a git repo. Define config client yaml and twitter to kafka yaml.

Create a spring cloud config server microservice by adding the config server module and setting server port 8888. Configure dependencies, bootstrap, and a git repository to enable config retrieval.

Centralize logging by using a common logback xml in the app config data module and including it in each service’s logback to share appenders, with dev home and name configurable.

Enable twitter-to-kafka to fetch remote config from spring cloud config server via bootstrap yaml and config-client, ensuring the config server and kafka cluster run before mvn install.

Learn how to configure a remote github repository for a Spring config server, switch to main branch, use personal access tokens, and run twitter_to_kafka with config server and kafka.

Add basic authentication to the config server by configuring spring security and updating bootstrap yaml with spring_cloud_user and spring_cloud_password for both services; encrypt passwords using jasypt and jce branches.

Implement jasypt in a Spring Boot app with the jasypt spring boot starter, encrypting secrets and managing the encrypter password via environment variables for runtime decryption.

Encrypt secrets with Java cryptography extension using the spring encrypt command, updating bootstrap yaml. Use config server endpoints to decrypt and set encrypt_key as an environment variable.

Compare Jasypt and JCE for encrypting secrets, highlighting PBKDF2, salt, and iteration count, and discuss symmetric vs asymmetric approaches, config server decryption, and SSL/TLS in production.

Build a config server docker image with the spring boot maven plugin, configure docker compose, enable health checks for the actuator endpoint, and run twitter-to-kafka and config server together.

Explore how a Kafka consumer reads events from topics and partitions, manages offsets, and joins consumer groups for scalable processing. Understand delivery semantics and Spring Kafka support for reliable consumption.

Learn to add a Kafka consumer module, configure Avro deserialization, and set consumer properties like group id, auto offset reset, batch listener, and concurrency in a Spring Boot microservice.

Build the Kafka to Elastic service with Spring Boot, Spring Kafka, and Avro, configure the config server, and implement a Twitter Kafka consumer to log messages.

Create an onAppStarted event handler annotated as an EventListener to verify Kafka topics are created for elastic. Start the listener container explicitly after checks and disable auto startup.

Explore Elasticsearch basics and how the open source search engine uses Apache Lucene, RESTful JSON APIs, dynamic mapping, inverted indexes, analyzers, and a powerful query DSL.

Set up a three-node elasticsearch cluster with docker compose, enabling quorum via master eligible nodes, discovery seeds, memory lock, and expose port 9200; update env file to set version.

Create the elastic-model module with spring boot data elasticsearch and lombok, implementing TwitterIndexModel as a twitter-index document driven by elastic-config data.

Add the elastic config module and dependencies version 7.9.1. Implement ElasticSearchConfig to expose a high level rest client bean using UriComponentsBuilder for host, port, schema, with timeouts.

Create an elastic index client to perform bulk indexing against Elasticsearch, wiring app config and elastic model dependencies, and implement an elastic index util for converting documents to index queries.

Learn to index with Elasticsearch repositories in Spring by configuring a conditional on property to switch between Twitter Elasticsearch index client implementations and compare repository-based approaches with low-level Elasticsearch queries.

Integrate elastic modules with the Kafka to Elastic service by adding dependencies, transforming Avro models to Elastic models, and persisting tweets to Elasticsearch using a transformer and Elastic index client.

Build and run the kafka to elastic service in docker compose, using spring boot maven plugin to create the image, configure depends_on, and validate topics with startup scripts.

Explore querying Elasticsearch with its DSL using Postman. Index data and practice key queries like match, term, keyword, query_string, and wildcard, plus size and scroll controls.

Develop an elastic query client module for the elastic query microservice to query Elasticsearch by ID, text, or all, using native and bool queries.

Leverage a spring data elasticsearch repository to query the twitter index by text and id, using a dedicated query client and a singleton collections util.

Builds the elastic query microservice as the CQRS query component, querying Elasticsearch and separating command and query paths in an event-sourced, Kafka-based microservices architecture.

Create a Spring Boot web controller under /documents with lombok-based request and response models, and secure endpoints using HTTP basic auth and bcrypt password encoding.

Create a Spring service layer to return elastic data in the controller by implementing ElasticQueryService, converting Twitter index models to response models with ElasticToResponseModelTransformer, and wiring through TwitterElasticQueryService and ElasticQueryClient.

Add spring boot validation and use NotEmpty and Valid to validate inputs, then implement a controller advice to handle AccessDeniedException, IllegalArgumentException, RuntimeException, and method argument validation errors.

Explore HATEOAS, hypermedia as the engine of application state, and see how REST responses use hypermedia links and HTTP verbs like GET, POST, PUT, DELETE, and PATCH.

Add hateoas support to the rest api with spring boot hateoas starter, and build an assembler to expose hypermedia links via toModel and toModels.

Learn API versioning for REST services, including semantic versioning, backward and forward compatibility, and when to introduce major versions. Compare URI, media type, headers, and query parameter versioning.

Implement api versioning for the Elastic query service using uri and media type approaches, adding v2 models, mappings, and hateoas links.

Apply open api v3 documentation to a spring boot elastic query service using springdoc-openapi-ui. Generate json, yaml, and html docs with swagger annotations and content negotiation.

Create the elastic query service docker image with the spring boot maven plugin, set environment variables (including port 8183), and run via docker-compose to test the API with postman.

Build a non blocking web client with Spring web client to access elastic query service, using Thymeleaf and Bootstrap for the frontend, and a client-side load balancer by Spring Boot.

Begin implementing the elastic-query-web-client module by adding dependencies, configuring a web client with timeouts and base URL, and setting up thymeleaf, security, and config server, plus a context load test.

Create Thymeleaf templates with bootstrap, place header, footer, and error pages in resources/templates, and build a home page with a search form using th:action, th:object, th:each, and th:switch.

Develop a Spring Boot web controller with validation for query-by-text, define elastic query request and response models, and integrate thymeleaf with error handling via a controller advice.

Adds security to the web client by configuring http basic authentication and in-memory bcrypt users, while laying groundwork for oauth and openid connect, plus a load-balanced web client builder.

Develop an elastic query web client with ElasticQueryWebClient and a Spring service, posting to get-document-by-text and returning response models for Thymeleaf display.

Add spring cloud load balancer and apply LoadBalanced to the web client, configuring elastic-query-service with two instances on ports 8183 and 8185, later replacing with discovery service and api gateway.

Create a docker image for the web client with spring boot maven plugin, run it via docker compose, and observe load balancing across two query service instances that call Elasticsearch.

Create reusable elastic query service and web client modules by building a common module with shared models, transformers, and controller advice, enabling dry reuse across reactive and non-reactive versions.

Learn to build a reactive elastic query service with Spring Boot WebFlux, reactive elastic repository, and Flux-based streaming, including back-pressure and Reactor Netty backend.

Develop a reactive elastic query web client using spring webflux and webclient, leveraging the elastic query web client common module, config server, and thymeleaf reactive templates to stream Flux results.