I am in the process of integrating OTEL traces/metrics/logs into our application stack. For the most part this process has been going good, with the exception of a few hiccups along the way which -- fortunately -- have been resolved. Currently our set of applications uses both blocking and reactive implementations, meaning that I need to integrate OTEL to instrument both.
For the case of the blocking apps, everything seems to be going smoothly and I have been able to create useful spans using the various instrumentation libraries like Mongo, WebMvc etc. This results in the following spans to be created:
As you can see the drilldown is quite nice and everything gets captured correctly.
However when I attempt to do the same for a non-blocking/reactive application I am getting inconsistent spans. For instance check the following:
As you can see the three spans:
- journey find sis.journeys
- journey POST /journeys
- journey http post /journeys
should have been one single span with no3 being the parent and 2 and 1 being child spans. I have tried to debug this and figure out what the problem really is but I am a bit lost. As far as I can tell I am not doing anything special in contrast to the blocking app, other than attempting to instrument a reactive app in this case.
Library wise I have added the following in place:
runtimeOnly 'io.micrometer:micrometer-registry-otlp:1.12.1'
implementation 'io.micrometer:micrometer-tracing-bridge-otel:1.2.1'
implementation 'io.opentelemetry:opentelemetry-exporter-logging'
implementation 'io.opentelemetry:opentelemetry-exporter-otlp'
runtimeOnly 'io.opentelemetry:opentelemetry-exporter-otlp-common'
implementation 'io.opentelemetry.instrumentation:opentelemetry-logback-appender-1.0'
implementation 'io.opentelemetry.instrumentation:opentelemetry-resources'
implementation 'io.opentelemetry.instrumentation:opentelemetry-mongo-3.1'
implementation 'io.opentelemetry.instrumentation:opentelemetry-spring-web-3.1'
implementation 'io.opentelemetry.instrumentation:opentelemetry-spring-webflux-5.3'
implementation 'io.opentelemetry.instrumentation:opentelemetry-instrumentation-annotations'
and as you can see I am bridging OTEL with micrometer. Based on all the above my questions are the following:
- How can I resolve the issue of properly instrumenting a reactive application and end up producing correct spans
- In both cases there is a span created by a library named
org.springframework.boot
. This is seemingly a duplicate span having the same information as the one captured by either the webmvc instrumentation library or the webflux instrumentation library. While not a problem, is there a way to disable this and what really produces this?
Edit: Adding result after swapping out of libraries
I can't see any error in our dependencies, so I would file this as a bug with spring-boot (which may end up in micrometer, that's a bit hard to know).
In the meantime, you can try out another spring starter, which uses different libraries, so it may not be affected by the bug: https://opentelemetry.io/docs/languages/java/automatic/spring-boot/
Note that this starter is not compatible with the starter from spring boot - configuration and produced telemetry differs.