OpenTelemetry Spring Boot Reactive incorrect spans

197 Views Asked by At

I am in the process of integrating OTEL traces/metrics/logs into our application stack. For the most part this process has been going good, with the exception of a few hiccups along the way which -- fortunately -- have been resolved. Currently our set of applications uses both blocking and reactive implementations, meaning that I need to integrate OTEL to instrument both.

For the case of the blocking apps, everything seems to be going smoothly and I have been able to create useful spans using the various instrumentation libraries like Mongo, WebMvc etc. This results in the following spans to be created:

enter image description here

As you can see the drilldown is quite nice and everything gets captured correctly.

However when I attempt to do the same for a non-blocking/reactive application I am getting inconsistent spans. For instance check the following:

enter image description here

As you can see the three spans:

  1. journey find sis.journeys
  2. journey POST /journeys
  3. journey http post /journeys

should have been one single span with no3 being the parent and 2 and 1 being child spans. I have tried to debug this and figure out what the problem really is but I am a bit lost. As far as I can tell I am not doing anything special in contrast to the blocking app, other than attempting to instrument a reactive app in this case.

Library wise I have added the following in place:

    runtimeOnly 'io.micrometer:micrometer-registry-otlp:1.12.1'
    implementation 'io.micrometer:micrometer-tracing-bridge-otel:1.2.1'

    implementation 'io.opentelemetry:opentelemetry-exporter-logging'
    implementation 'io.opentelemetry:opentelemetry-exporter-otlp'
    runtimeOnly 'io.opentelemetry:opentelemetry-exporter-otlp-common'
    implementation 'io.opentelemetry.instrumentation:opentelemetry-logback-appender-1.0'
    implementation 'io.opentelemetry.instrumentation:opentelemetry-resources'
    implementation 'io.opentelemetry.instrumentation:opentelemetry-mongo-3.1'
    implementation 'io.opentelemetry.instrumentation:opentelemetry-spring-web-3.1'
    implementation 'io.opentelemetry.instrumentation:opentelemetry-spring-webflux-5.3'
    implementation 'io.opentelemetry.instrumentation:opentelemetry-instrumentation-annotations'

and as you can see I am bridging OTEL with micrometer. Based on all the above my questions are the following:

  1. How can I resolve the issue of properly instrumenting a reactive application and end up producing correct spans
  2. In both cases there is a span created by a library named org.springframework.boot. This is seemingly a duplicate span having the same information as the one captured by either the webmvc instrumentation library or the webflux instrumentation library. While not a problem, is there a way to disable this and what really produces this?

Edit: Adding result after swapping out of libraries

enter image description here

1

There are 1 best solutions below

6
On

I can't see any error in our dependencies, so I would file this as a bug with spring-boot (which may end up in micrometer, that's a bit hard to know).

In the meantime, you can try out another spring starter, which uses different libraries, so it may not be affected by the bug: https://opentelemetry.io/docs/languages/java/automatic/spring-boot/

Note that this starter is not compatible with the starter from spring boot - configuration and produced telemetry differs.