OpenTelemetry Observability For Java Spring Boot Developers

Differentiate observability from monitoring to reveal why a system is slow. Use traces and internal visibility to locate root causes and guide fixes.

Define observability as understanding the system’s internal state through signals. Explain the three pillars: metrics, logs, and traces, and show how traces reveal end-to-end request flows in a distributed system.

OpenTelemetry provides a vendor-neutral standard for logs, metrics, and traces, decoupling applications from observability backends and supporting polyglot languages and multiple tools via a simple collector.

The lecture contrasts micrometer with OpenTelemetry, showing OpenTelemetry as a full observability standard with distributed tracing that prevents fragmented, polyglot microservice monitoring.

Learn how instrumentation enables observability by emitting telemetry signals, comparing automatic (zero-code) instrumentation with agents and manual code changes for deep business visibility.

Explore observability with opentelemetry in the trace flicks microservices demo. Learn how movie, actor, and review services interact via distributed tracing, metrics, and dto-based api responses.

Follow a hands-on TraceFlix walkthrough showing Spring bean configuration, movie dto construction with actor and review data, and intentional slow processing to illustrate observability.

Walks through building and running the TraceFlix microservices with Docker Compose, including per-service dockerfiles, a Maven multi-module project, and inter-service calls via service names to observe traces, metrics, and logs.

Build and run the Spring Boot app with maven and docker compose, test the movies API via curl or postman, observe warmup timing and 500 errors, then enable telemetry.

OpenTelemetry enables observability with standard APIs, SDKs, and tools for telemetry. Learn monitoring vs observability, signals (logs, metrics, traces), and auto vs manual instrumentation.

Learn how distributed tracing with OpenTelemetry tracks the complete journey of a request across a microservices architecture, using traces and spans to understand the flow and identify issue owners.

Attach a runtime agent to the JVM to enable OpenTelemetry automatic instrumentation for distributed tracing in the Trace Flex app, configured via environment variables without editing the source code.

Explore OpenTelemetry architecture, where a Java agent instruments requests at runtime and forwards telemetry to the collector, which exports traces and metrics to observability backends.

Enable distributed tracing across movie, actor, and review services by configuring the OpenTelemetry Java agent in a Docker Compose setup, with the Otel collector, Tempo, and Grafana for visualization.

Configure the otel collector to receive traces via grpc 4317 or http 4318 and forward them to tempo, using docker compose with Grafana for visualization.

Spin up six containers with docker compose for distributed tracing, verify they run, then access Grafana and configure Tempo as the data source.

Explore how distributed tracing records traces and spans across actor, movie, and review services, revealing database queries and inter-service calls to diagnose n+1 issues in microservices.

Examine how spans form a root and child hierarchy within a trace and how the Java agent uses trace parent headers to link concurrent requests across services and queries.

Clarify how tracing works by defining a trace as a set of spans and their parent-child relationships, and emphasize why this relationship matters.

Explore resource attributes and span attributes in OpenTelemetry. Resource attributes describe the service or container, while span attributes cover specific operations like HTTP methods and database statements.

Demonstrates distributed tracing to diagnose upstream failures in a microservices flow, revealing which service causes delays and errors with root spans, events, and HTTP 500/404 failures.

Understand the Maven Spring Boot project structure, build the jar and Docker images, and learn that code updates require mvn clean package and Docker Compose build to reflect in containers.

Demonstrates capturing and logging the traceparent header in movie and review services, builds docker images, runs the distributed tracing demo with docker compose, and verifies trace details via postman.

Learn to search and filter Grafana spans by service, status, tags, and http path, then create dashboards for slow or failed traces.

Troubleshoot Grafana tag search issues and the search view, compare with T-sql, and use trace attributes and span attribute names and values in Trace Shell with postman checks.

Examine the limitations of automatic instrumentation when issuing concurrent io requests in a Java Spring Boot service. Implement stream gatherers and virtual threads, noting how thread locals break trace relationships.

Revert the stream gatherers changes in the service class, stop running docker containers, build the jar with Maven clean package, and rebuild the docker images with docker compose build.

Enable distributed tracing with automatic instrumentation via the OpenTelemetry Java agent, and use the Otel collector to export signals with W3C trace context, spans, and attributes.

Explore OpenTelemetry sampling strategies to control trace volume and costs, including head and tail sampling, always on, always off, trace id ratio, and parent-based approaches.

Explore the always on strategy to trace all incoming and outgoing requests in a development environment for distributed tracing with Docker Compose, actor 8081, review 8082, Postman, and Grafana.

Apply the always off strategy to disable tracing in a dockerized Spring Boot observability setup, showing no traces in Tempo or Grafana when requests run.

Use the trace ID ratio strategy to sample a fixed percentage of requests. Configure the percentage with a property (0.5 equals 50%), and verify with docker compose, postman, and Grafana.

Explain the parent based tracing strategy, using the trace parent header and trace flag, with always on, always off, and trace id ratio configurations across movie, review, and actor services.

Explore parent-based always-on tracing in OpenTelemetry for Java Spring Boot, showing docker compose setup, trace id ratio, and service-specific overrides between movie, actor, and review services.

Explore postman's collection run feature to set iterations and automatically send multiple requests, supporting observability workflows for Java Spring Boot development.

Explore the parent based always off strategy, with the movie service at a 20% trace ratio while actor and review services stay off, and observe traces in grafana tempo.

Explore the parent based trace ID ratio in OpenTelemetry observability for Java Spring Boot, configuring movie service at 20% and actor service at 50% to govern tracing.

Explore tail sampling in OpenTelemetry: collect all spans, let the collector decide what to export to tempo, favoring failed or slow requests for analysis.

Configure tail sampling by status code to keep failed traces. Use the collector and processor, cap memory at 100,000 traces and a 10-second wait.

Apply latency-based tail sampling to capture only slow requests, with a 1500 millisecond threshold, and visualize traces in Grafana from a docker-compose setup.

Configure tail sampling to capture slow and failed requests using the status code and latency policies; copy and paste the example configurations to enable precise monitoring.

Apply tail sampling by filtering traces with a span attribute, such as the URL path API movies two, and validate the setup using Docker Compose, Postman requests, and Grafana.

Clarifies why URL path based sampling is tail sampling, contrasts attribute-based sampling with head sampling, notes no built-in head sampler, and previews a custom sampler later in the course.

Explore sampling strategies for tracing in different environments, including head and tail sampling, always on/off, trace ID ratio, and policies based on status, latency, or attributes.

Learn how metrics provide lightweight, time-based measurements that reveal request rate, average response time, CPU and memory usage, and how many requests are failing to show the system's overall health.

Explore zero code metrics with the OpenTelemetry Java agent, sending JVM, GC, thread, cpu usage, and http/grpc metrics to the Otel collector and Prometheus.

Discover why PromQL is essential for querying Prometheus metrics and how it differs from SQL-like queries. The lesson introduces the basics of Prometheus query language and its role with Grafana.

Prometheus is a lightweight time-series database that collects and queries metrics from microservices using a pull-based model, with service discovery, labels, and an http-based querying interface.

Prometheus targets operational signals and aggregated metrics, not business data accuracy, and scraping every 15 seconds may miss metrics during crashes, with rate increases and approximations revealing trends.

Set up a matrix demo project to send metrics to the Otel collector using Prometheus. Explore docker compose changes, the Otel collector config, and Grafana integration for Prometheus scraping.

Demonstrates end-to-end observability in a metrics demo for Java Spring Boot, using Prometheus, Docker Compose, and otel collector to monitor server and http client metrics.

Explore how metrics are queried and visualized with instant and range vectors, showing how labels group movie, review, and actor service requests and track status codes like 200 and 404.

Explore basic PromQL queries in Prometheus, filtering by the exported job to focus on movie service entries, and refine results with status codes, regex matching, and not equals.

Use PromQL aggregations to compute total requests and failures by service or http route, employing sum by and filters on http status codes to explore per-endpoint metrics.

Explore how Prometheus range functions turn counter values into actionable metrics, such as error rate and failures per minute, rather than raw totals, and query last 24 hours.

Learn to use Prometheus range vectors to measure movie service requests in the last minute, applying increase and rate functions and understanding when windows yield empty results.

Learn to compute average response time in Prometheus by using rate over the last minute, comparing total time and count, with practical postman examples showing endpoint performance.

Prometheus range functions handle counter resets by comparing pre- and post-reset values, ensuring rate and increase yield accurate metrics after restarts.

Explore open telemetry observability for java spring boot developers by examining metrics beyond http server data, including database connections and memory usage, with grafana dashboards.

Connect Grafana to Prometheus to build dashboards tracking CPU utilization and movie service throughput, and visualize requests between movie, actor, and review services.

Explore metrics as aggregated numerical measurements over time, not traces, and learn when to use counters, gauges, or histograms with automatic and manual instrumentation in Prometheus.

Explore how logs explain why decisions were made by capturing time-stamped, structured or semi-structured data across microservices, and centralize them to debug a single request across threads.

Improve logging and error handling in trace flex app to export data with problem detail object for actor and movie service failures, and copy maven project to a logs directory.

Implement custom actor not found exception handling with a controller advice to return a problem detail 404, and log using SLF4J in a Spring Boot service.

Explain the problem detail RFC 7807/9457 standard for error responses, outlining type, title, status, detail, and instance, with human and machine readable formats and practical examples.

Implement logging of headers and movie id, and robust error handling across movie, review, and actor services, including http client errors, problem details, and not found or resource access exceptions.

Demonstrates updating the trace flex application, building docker images with maven and docker compose, and testing movie service responses from Postman, including 404 for missing data and error detail responses.

Add the OpenTelemetry logback appender dependency in your Maven project and update logback.xml to bridge your logging framework with the opentelemetry collector, enabling logs to flow to loki.

Manage dependency versions in the Trace Flex pom.xml via a shared property and add a Logback.xml for each module's appender, then build jars and docker images.

Set up logs infrastructure using docker-compose with tempo and Loki, configure Opentelemetry Appender, and link traces and logs in Grafana for seamless cross-navigation.

Set up Grafana with Loki and temple data sources, then explore logs and traces, build queries, and create dashboards showing inter-service activity from movie, review, and actor services.

Explore logs and distributed tracing in Java Spring Boot by filtering trace IDs, viewing error logs, and diagnosing service unavailable and not found errors across movie, review, and actor services.

Edit your dashboard in edit mode, save changes, and export the JSON or copy it to the clipboard; paste the JSON into Grafana and restart to load the updated dashboard.

Enable and link traces and logs in your dockerized OpenTelemetry setup, view distributed tracing in Grafana and Tempo, and inspect log events for a specific trace ID.

Export logs from the Java Spring Boot app to the OpenTelemetry collector, add an appender, and enrich logs with trace information to filter by trace ID in Grafana.

Explore manual instrumentation with the OpenTelemetry API by building a Maven project, creating spans and metrics via providers, exporting to a collector, and visualizing in Grafana using Docker Compose.

Set up a Maven project for OpenTelemetry, copy dependencies and dependency management into pom.xml, include OpenTelemetry API SDK and gRPC exporter with Logback, then create an OpenTelemetry config class.

Configure an OpenTelemetry instance with a tracer provider and exporter, using a gRPC span exporter to localhost:4317, and choose a simple span processor for local development.

Explain why attribute keys encode type to avoid casting when accessing attributes. Show how to build a reusable OpenTelemetry instance with a tracer provider and follow the discussed naming conventions.

Demonstrate distributed tracing with manual instrumentation by simulating an order service that calls payment, inventory, and notification services, generating observable spans in Grafana through deliberate delays.

Create spans with an OpenTelemetry tracer using an instrumentation scope name to identify libraries and versions. Grafana visualizes service names, library versions, and span attributes for debugging.

Explore why mixing telemetry with business logic appears in a framework-free setup while wiring OpenTelemetry spans manually, revealing the API before Spring Boot simplifications.

Create a span manually with a tracer and builder. Start the span, perform business logic, and end it to export span details to the OpenTelemetry collector via the gRPC exporter.

Set up the observability infrastructure to export spans from the manual instrumentation demo to the OpenTelemetry collector, visualize traces in Tempo via Grafana, and explore span attributes.

Learn to end spans reliably with try-catch and finally, set span status to ok or error, and record exceptions for Grafana traces in OpenTelemetry for Java Spring Boot.

Set span attributes with key-value pairs and maps, view them in Grafana, and filter traces by order id and amount to diagnose order processing.

Create multiple spans to form a parent-child relationship between process order and process payment, keeping telemetry logic separate via aspect oriented programming.

This lecture shows how OpenTelemetry context and context storage enable automatic parent-child relationships between spans via thread-local storage, with make current updating context and ensuring proper cleanup.

Learn to create a parent-child span relationship in OpenTelemetry, using try-with-resources scope to set process order as the parent of process payment, and view traces in Grafana.

Establish nested spans with a parent-child relationship by linking process order, process payment, deduct inventory, and send notification, ensuring each call uses the previous span as its parent.

Demonstrates establishing explicit parent-child relationships between spans in OpenTelemetry for Java Spring Boot, using context.current and set_parent to control span hierarchy across threads.

Explore explicit parent-child spans across multiple threads by attaching inventory and notification spans to separate threads, including platform and virtual threads, and observe parallel versus sequential traces in Grafana.

Apply the execute around method pattern to remove tracing boilerplate by moving setup and teardown into a trace utility and passing business logic as a callback.

Refactor the trace using trace util in the trace package to create spans for send notification, detect inventory, process payment, and process order, with telemetry attributes visible in Grafana.

Learn how to add span events to traces, with a payment retry example showing how retries reveal timing and errors in distributed tracing via Grafana.

Explore asynchronous context propagation and its impact on trace parent-child relationships when running work on separate threads. Compare using span.makeCurrent with thread-local storage and explicit span passing as solutions.

Shows how OpenTelemetry context propagation works by wrapping runnables to capture and reuse the current context, establishing parent-child relationships and tracing outcomes in Grafana.

Learn how to set span kind in OpenTelemetry spans to categorize work as server, client, producer, consumer, or internal, with internal as the default, and view the value in Grafana.

Configure the OpenTelemetry meter provider with a grpc metric exporter and a periodic metric reader, exporting metrics at five-second intervals for local demos.

Explore OpenTelemetry metrics naming: compose names with namespace, entity, operation, and unit; use app-like patterns, with examples like http server request duration and memory utilization.

Create metric counter with meter counter builder, name it, describe it, and set a unit. Use dimensionless counts or pseudo units like {request_count} and apply duration, bytes, or percentage units.

Show how to implement an OpenTelemetry product view counter in a Java Spring Boot app, wiring a shared meter into a product controller and incrementing on every view.

Product view counter demo for Java Spring Boot shows exporting metrics via OpenTelemetry collector to Prometheus (scraped every 10s) and visualizing through Grafana, using rate and increase to measure throughput.

Explain why metric names differ in Prometheus when sending metrics to the OpenTelemetry collector, noting that Prometheus replaces dots with underscores and appends total for counters, which is normal.

Introduce a dedicated product view recorder to track views by product id using attributes. Inject it via constructor as a spring component and call record view at the view endpoint.

Demonstrates OpenTelemetry observability with a counter with attributes to track product view counts in a Java Spring Boot app, visualizing with Prometheus and summing to total product views.

Implement a long gauge metric to track JVM memory used, using a gauge builder with a callback that computes runtime memory usage and reports to the Otel collector.

Run the observability stack, monitor JVM memory with Prometheus, and analyze the sawtooth memory pattern caused by garbage collection, including startup memory and allocation behavior.

Explore how the up down counter records increments and decrements over time, not fixed values like a gauge, and apply it to track sessions, queues, and connections with OpenTelemetry collector.

Demonstrates implementing an up down counter to track active tasks by wrapping runnables, incrementing on start and decrementing on completion, with a virtual-thread per task executor demo.

Use an up down counter demo to monitor active tasks with Prometheus UI, viewing the task active count metric and graph over time to track active sessions.

Track task duration distributions with histogram data in Prometheus, exposing bucket, sum, and count metrics to enable percentile calculations and fast averages.

Create a long histogram named app.task.execution.duration to measure task execution time in milliseconds, with buckets from 1000 to 10000 ms.

Run the histogram demo to monitor task executor latency with Prometheus metrics, explore bucket distributions, and compute 90th and 99th percentiles plus average response time using read and increase.

Explore how the mapped diagnostic context (MDC) enriches log statements with contextual data like user id in multi-user production environments, enabling log correlation across services via thread-local context.

Configure the OpenTelemetry logger provider and batch log record processor, connect the appender via logback XML, and capture MDC attributes to send logs to the collector.

Export simple logs to Loki using the OpenTelemetry appender and an MDC demo in a payment service that logs payment started and payment processed.

Demonstrates using MDC to attach user id and other attributes to logs across console and Grafana, including single key, multiple keys, and map contexts with automatic cleanup.

Explore MDC context propagation across threads in Java, using a copy of the MDC context map to transfer attributes to child threads so logs show the user id.

Turbo filter enables debug logging dynamically for that single request or transaction, balancing production performance with diagnosing specific issues without noisy global logging.

Demonstrates building a Logback turbo filter to enable debug logs for specific requests by checking MDC user id, configuring with Logback XML, and routing to Otel collector and Grafana Loki.

Clarify turbo filter performance by testing rather than assuming impact. Keep the logic simple, because the MDC in-memory lookup is lightweight, while database or network calls can change outcomes.

Learn manual instrumentation with OpenTelemetry by creating tracer, meter, and logger providers to emit traces, spans, and metrics that export to the collector, with context propagation and MDC logging.

Auto-configure OpenTelemetry with Spring Boot using the OpenTelemetry Spring Boot Starter package, auto-configuring tracing, metrics, and logging, enabling zero code instrumentation while allowing custom spans, developed by the OpenTelemetry community.

Set up the trace flex maven project structure, apply shared dependency management and starter package across modules, and update pom.xmls with the hotel version for actor, movie, and review services.

Disable the OpenTelemetry sdk for local development by setting sdk.disabled to true or using a local profile, then rebuild jars and docker images to run without collector errors.

Explore production observability with OpenTelemetry in a multi-service docker-compose setup. Configure collector, Tempo, Loki, and Prometheus, and view signals in Grafana dashboards.

Enable sdk for local development to test observability using ide or docker compose, managing observability properties across three services and including hotel properties in application.properties.

Set up a zero code instrumentation demo with a docker-compose environment, enable OpenTelemetry, and observe traces, metrics, and logs in Grafana dashboards without changing code.

Explore zero code instrumentation to observe logs, traces, and metrics, validate throughput on the metrics dashboard, and use Grafana queries to analyze last minute per-service requests and durations.

Demonstrate fixing concurrent requests in a Java Spring Boot app by using OpenTelemetry context wrap to propagate tracing context across threads, enabling linked spans in Grafana traces.

OpenTelemetry observability for Java Spring Boot developers guides you to implement custom spans, using the span builder and span.current to add events and attributes for movie details.

Learn to implement custom metrics in a Java Spring Boot app by tracking the top three movie views with an OpenTelemetry counter, exporting to a collector, and querying Prometheus.

Demonstrate custom metrics in OpenTelemetry for Java Spring Boot by running Docker Compose and using Prometheus to identify top movies via increase, sum by movie id, and top k.

Customize the opentelemetry auto configuration in Java Spring Boot apps by adjusting tracer and meter providers, exporters, and metric reader, with property-first configuration.

Discover how the OpenTelemetry Spring Boot starter auto-configures providers and the OpenTelemetry object, enabling zero-code instrumentation to export traces, metrics, and logs, and share context for tracing with custom spans.

Cultivate a staff engineer mindset to design on-demand observability that enables real-time debugging and tracing across services, empowering teams with platform-wide insights without redeploys.

Define goals to enable per-request distributed tracing and detailed logging across thousands of microservices, using a reusable module and a controllable trace ID for production debugging.

Describe a high level design to enable tracing and detailed logging in a java spring boot app using a debug request header, text map propagator, MDC, and turbo filter.

OpenTelemetry baggage acts as a context-based key value store that propagates across services, automatically transferring data via http headers, grpc metadata, or kafka message headers.

Avoid storing sensitive information in baggage and keep it for observability-related requirements. Be cautious when services call third-party APIs to prevent exposing sensitive data.

Set up the on demand observability module in the trace flex maven project, add OpenTelemetry Spring Boot starter and spring web as provided dependencies, and configure the four packages.

Implement a text map propagator for OpenTelemetry to extract and inject the debug request header, and use baggage to store the request ID in the immutable context.

The text map propagator looks for a debug request header, stores it in baggage, and returns an updated context. Beyond the first service, propagation relies on baggage and OpenTelemetry.

Implement an on-demand trace sampler that uses header presence to decide tracing, otherwise applying a 10% trace id ratio. It delegates to the default and reads baggage for request id.

Create a baggage MDC bridge filter that extends the once-per-request spring web filter, accesses baggage from the context, extracts the request id, and sets it in MDC for downstream logs.

Implement a turbo filter that uses the MDC to detect a debug request by checking the request id, enabling detailed logging for debug requests and default logging otherwise.

Programmatically register a reusable turbo filter for spring boot apps during the application ready event, wiring it into the logback logger context. Check for existing filters to avoid duplicate registration.

Expose spring beans via on demand observability configuration, wiring baggage MTC bridge filter and log filter register beans, and customize samplers and text map propagators with auto configuration customizers.

Define an on demand observability annotation with runtime retention and class-level target, import the config, and enable it in the Spring Boot main class so Spring auto-creates observability beans.

Add the reusable module as a dependency and enable it with an annotation. Configure logback to capture MDC attributes and manage the module version in the root pom.xml.

Prepare for the demo by building jar files, then start the observability infrastructure with docker compose in the zero eight directory, and verify containers with docker ps -a.

Demonstrate on-demand observability by enabling distributed tracing, detailed logging, and per-request log level control with trace IDs, adjustable sampling, and centralized viewing in Grafana.