Spring Boot 3 WebFlux Observability Best Practices¶
Author: Liu Rui
How to use Micrometer for Distributed Tracing in Spring Boot 3 WebFlux with Reactive Kotlin
Distributed tracing is a very useful tool in an observability software system. It enables developers to understand when, where, and how different interactions occur within and between applications, making it easier to observe complex software systems.
Starting from Spring Boot 3, the old Spring Cloud Sleuth solution for distributed tracing in Spring Boot has been replaced by the new Micrometer Tracing library.
You may already be familiar with Micrometer, as it was previously used as the default solution for exposing platform-independent metrics and monitoring JVM-based microservices (e.g., Prometheus). The latest product extends the Micrometer ecosystem with a platform-independent distributed tracing solution. This allows developers to use a common API to instrument their applications and export tracing data in different formats to tracing collectors such as Jaeger, Zipkin, or OpenTelemetry.
1. Microservice Setup¶
Next, we will create a simple Spring Boot microservice that provides a reactive REST endpoint, which internally queries a third-party service to retrieve some information. The goal is to export traces for the two operations.
We will start with the following Spring Boot Initializr project, which you can find here. It includes Spring Boot 3.0.1 with Kotlin Gradle DSL, Spring Web Reactive (WebFlux), and Spring Actuator with Prometheus. The following code primarily uses Kotlin, but it is also possible with Java, and most methods are the same.
Define the Endpoint¶
We will start by adding a simple REST controller class with a test endpoint that calls an external API using Spring WebClient. We use the suspend keyword to leverage Kotlin coroutines. This allows us to write imperative code while taking advantage of Spring WebFlux's reactive streams.
In the following example, we use Spring WebClient to call an external TODO-API, which returns a TODO item as a JSON string. We will also create a log message that should later contain some tracing information.
@RestController
class Controller {
val log = LoggerFactory.getLogger(javaClass)
val webClient = WebClient.builder()
.baseUrl("https://jsonplaceholder.typicode.com")
.build()
@GetMapping("/test")
suspend fun test(): String {
// simulate some complex calculation
delay(1.seconds)
log.info("test log with tracing info")
// make web client call to external API
val externalTodos = webClient.get()
.uri("/todos/1")
.retrieve()
.bodyToMono(String::class.java)
.awaitSingle()
return externalTodos
}
}
Add Micrometer Tracing¶
In the next step, we add the Micrometer Tracing dependency to our build.gradle.kts file. Since Micrometer supports different tracing formats and vendors, the dependencies are separated, and we only import what we need. To keep all dependencies synchronized, we use the Micrometer Tracing BOM (Bill of Materials). Additionally, we add the core dependency and a bridge to convert Micrometer Tracing to the OpenTelemetry format (other formats are also available).
implementation(platform("io.micrometer:micrometer-tracing-bom:1.0.0"))
implementation("io.micrometer:micrometer-tracing")
implementation("io.micrometer:micrometer-tracing-bridge-otel")
We also need to add an exporter dependency to export the created traces. In this example, we will use the Zipkin exporter maintained by OpenTelemetry and supported by Micrometer Tracing.
Configuration¶
Configuration is an essential step for setting up tracing. The configuration file application.yaml is located in the src/main/resources directory.
First, we must enable tracing in the management settings. We also set the tracing sampling probability to 1 (the default is 0.1) so that a trace is created for every call the service receives. In a production system with a high volume of requests, you may only want to sample a subset of traces. Additionally, we can define the endpoint URL to which the Zipkin exporter should send traces. Finally, we must update the default logging pattern to include the trace ID and span ID.
management:
tracing:
enabled: true
sampling.probability: 1.0
zipkin.tracing.endpoint: http://localhost:9411/api/v2/spans
logging.pattern.level: "trace_id=%mdc{traceId} span_id=%mdc{spanId} trace_flags=%mdc{traceFlags} %p"
2. Testing¶
Now that we have completed the service setup, we can run it. When you start the application, the server should start on port 8080 by default. You can then invoke the endpoint we created by opening a browser at http://localhost:8080/test. The response content is as follows:
To view the actual traces created when the endpoint is called, we need to collect and inspect them. In this tutorial, we will use the Zipkin exporter to export data to Guance. Of course, other systems such as Zipkin, Grafana Loki, or Datadog can also be used.
Now you can call the endpoint of our Spring Boot service again. After that, when you search for any trace in Guance, you should be able to find the tracing information for the endpoint request.
3. Issues¶
At first glance, everything seems to work fine. However, we have two issues.
Some of these issues have been addressed in the Micrometer Tracing documentation.
Missing Data in Logs¶
If we look at the application logs, we can see the log message emitted when the endpoint is called.
trace_id= span_id= trace_flags= INFO 43636 --- [DefaultExecutor] com.example.tracing.Controller : test log with tracing info
As you can see, the trace_id and span_id are not set. This is because Micrometer Tracing cannot yet easily handle the tracing context in reactive streams. Additionally, the Kotlin coroutine wrapper for reactive streams hides the tracing context. Therefore, we must defer the context of the current reactive stream to obtain the tracing information. In practice, this looks as follows:
Mono.deferContextual { contextView ->
ContextSnapshot.setThreadLocalsFrom(
contextView,
ObservationThreadLocalAccessor.KEY
).use {
log.info("test log with tracing info")
Mono.empty<String>()
}
}.awaitSingleOrNull()
To make it more reusable, we can extract the sample code into a separate function.
@GetMapping("/test")
suspend fun test(): String {
// ...
observeCtx { log.info("test log with tracing info") }
// ...
}
suspend inline fun observeCtx(crossinline f: () -> Unit) {
Mono.deferContextual { contextView ->
ContextSnapshot.setThreadLocalsFrom(
contextView,
ObservationThreadLocalAccessor.KEY
).use {
f()
Mono.empty<Unit>()
}
}.awaitSingleOrNull()
}
If we now start the application and call our endpoint, we should be able to see the trace_id in the logs.
trace_id=6c0053eba01199f194f5f76ff8d61917 span_id=967d591266756905 trace_flags= INFO 45139 --- [DefaultExecutor] com.example.tracing.Controller : test log with tracing info
WebClient Call Not Producing Traces¶
The second issue can be discovered by looking at the trace in Guance. It only shows the parent trace of the endpoint, but not the child span for the WebClient call. In theory, Spring WebClient, as well as RestTemplate, are automatically instrumented by Micrometer. However, if we look at the code, we are using the static builder method WebClient.builder(). To get automatic tracing from WebClient, we need to use the builder bean provided by the Spring framework. It can be injected via the constructor of our Controller class.
@RestController
class Controller(
webClientBuilder: WebClient.Builder
) {
val webClient = webClientBuilder // use injected builder
.baseUrl("https://jsonplaceholder.typicode.com")
.build()
// ...
}
After adjusting the code as above, when we call the endpoint again, we can see the WebClient span in Guance. Micrometer Tracing will also automatically propagate the trace_id in HTTP headers. For example, if we call another microservice that also has tracing enabled, it can receive the ID and send additional information to Guance.
4. Observability Guide¶
Micrometer Tracing automatically does a lot for us in Spring. However, sometimes we may want to add specific information to a trace span or observe specific parts of an application that are not incoming or outgoing calls.
Adding Span Tags¶
We can define custom tags and add them to the current observation to enrich the tracing data. To retrieve the current observation, we can use the ObservationRegistry bean. Similar to the logging issue, we must use the wrapper function to obtain the correct context.
@GetMapping("/test")
suspend fun test(): String {
observeCtx {
val currentObservation = observationRegistry.currentObservation
currentObservation?.highCardinalityKeyValue("test_key", "test sample value")
}
// ...
}
After adding this code, we can see our custom tag and its value in Guance.
Custom Observations¶
Creating custom observations (spans) using the Micrometer API is usually straightforward. However, when using reactive streams and coroutines, we need help with context propagation. If we create a new observation inside the endpoint handler, it will be treated as a separate trace. To make the code reusable, we can write a simple wrapper function that creates a new observation. It works similarly to the wrapper we created earlier for using the trace_id.
suspend fun runObserved(
name: String,
observationRegistry: ObservationRegistry,
f: suspend () -> Unit
) {
Mono.deferContextual { contextView ->
ContextSnapshot.setThreadLocalsFrom(
contextView,
ObservationThreadLocalAccessor.KEY
).use {
val observation = Observation.start(name, observationRegistry)
Mono.just(observation).flatMap {
mono { f() }
}.doOnError {
observation.error(it)
observation.stop()
}.doOnSuccess {
observation.stop()
}
}
}.awaitSingleOrNull()
}
This function can wrap any suspend function inside a new observation. It will automatically stop the observation once the given function is executed. Additionally, we track any errors that may occur and attach them to the trace.
We can now apply this function to observe any code, such as the execution of the delay function.
@GetMapping("/test")
suspend fun test(): String {
runObserved("delay", observationRegistry) {
delay(1.seconds)
}
// ....
}
After adding this code to the endpoint handler, Guance will show us a custom span for this operation.
5. Database Tracing¶
A typical Spring Boot application usually connects to a database in a real-world application. To leverage the reactive technology stack, it is recommended to use the R2DBC API instead of JDBC.
Since Micrometer Tracing is a relatively new technology, there is no automatic tracing available yet. However, the Spring team is working on creating auto-configuration. The experimental repository can be found here.
For the current project, we need to add the following dependencies to build.gradle.kts. For ease of testing, we will not use a real database but instead use the H2 in-memory database.
implementation("org.springframework.boot:spring-boot-starter-data-r2dbc")
runtimeOnly("com.h2database:h2")
runtimeOnly("io.r2dbc:r2dbc-h2")
// R2DBC micrometer auto tracing
implementation("org.springframework.experimental:r2dbc-micrometer-spring-boot:1.0.2")
In the Kotlin code, we add a simple CRUD repository with coroutine support. As shown below:
@Table("todo")
data class ToDo(
@Id
val id: Long = 0,
val title: String,
)
interface ToDoRepository : CoroutineCrudRepository<ToDo, Long>
@RestController
class Controller(
val todoRepo: ToDoRepository,
// ...
) {
@GetMapping("/test")
suspend fun test(): String {
// ...
// save
val entry = ToDo(0,"Springboot3 + WebFlux + Kotlin ")
todoRepo.save(entry)
// Sample traced DB call
val dbtodos = todoRepo.findAll().toList()
// ...
return "${dbtodos.size} $externalTodos"
}
}
Calling our endpoint will add one more span. The new span is named query and contains multiple tags, including the SQL query executed by Spring Data R2DBC.
Conclusion¶
Micrometer and the new tracing extension unify the observability technology stack for Spring Boot 3 and above. It provides a great abstraction for the different tracing solutions used by various companies and their technology stacks. Therefore, it simplifies the work for us developers.
In terms of reactive programming with Spring WebFlux, there is still room for improvement, especially with Kotlin. The Micrometer team is in active discussions with the team behind Project Reactor (the reactive library used by Spring WebFlux) to simplify the use of Micrometer Tracing with the reactive technology stack.