Tracing Spring Boot Micro services with Jaeger deployed on AKS

Question

I had setup Jaeger in Azure Kubernetes Cluster in monitoring namespace and I deployed my container which is instrumented with jaeger client libraries in monitoring domain. The service is up and running and I'm able to see the traces using actuator when I specify the :/actuator in the browser. But the same microservice is not populating in the service dropdown in Jaeger UI.

Below are the files i'm using.

DemoOpentracingApplication.java

        @SpringBootApplication
        public class DemoOpentracingApplication {
            @Bean
            public RestTemplate restTemplate(RestTemplateBuilder restTemplateBuilder) {

                return restTemplateBuilder.build();
            }

            @Bean
            public io.opentracing.Tracer jaegerTracer() {
                return new Configuration("spribng-boot", new Configuration.SamplerConfiguration(ProbabilisticSampler.TYPE, 1),
                        new Configuration.ReporterConfiguration()).getTracer();
            }

            public static void main(String[] args) {
                SpringApplication.run(DemoOpentracingApplication.class, args);
            }
        }


    HelloController.java

    @RestController
    public class HelloController {

        @Autowired
        private RestTemplate restTemplate;


        //private final Counter totalRequests= Counter.build().name("requests_total").help("Total Number of Requests").register();

        @Timed(
                value= "prometheus.hello.request",
                histogram=true,
                percentiles= {0.95,0.99},
                extraTags= {"version","1.0"}
                )
        @RequestMapping("/hello")
        public String hello() {

            return ("Hello From OPenTracing Controller");
        }

        @Timed(
                value= "prometheus.chain.request",
                histogram=true,
                percentiles= {0.95,0.99},
                extraTags= {"version","1.0"}
                )
        @RequestMapping("/chaining")
        public String chaining() {
            ResponseEntity<String> response = restTemplate.getForEntity("http://localhost:8080/hello",String.class);
            return "Chaining+" + response.getBody();
        }
    }


POM.xml
.....
<dependency>
            <groupId>io.opentracing.contrib</groupId>
            <artifactId>opentracing-spring-web-autoconfigure</artifactId>
            <version>0.0.4</version>
        </dependency>
....

Why the instrumented service is not populating in Jaeger UI in Kubernetes?

Answer 1

You can turn on the debugging in the client by setting the option JAEGER_REPORTER_LOG_SPANS to true (or use the related option in the ReporterConfiguration, as it seems that's how you are using it).

https://www.jaegertracing.io/docs/1.8/client-features/

Once you confirm the traces are being generated and sent to the agent, set the log-level in the agent to debug:

docker run -p... jaegertracing/jaeger-agent:1.8 --log-level=debug

If you don't see anything in the logs there indicating that the agent received a span (or span batch), then you might need to configure your client with the Agent's address (JAEGER_AGENT_HOST and JAEGER_AGENT_PORT, or related options in the Configuration object).

You mentioned that you are deploying in Azure AKS, so, I guess that the agent isn't available at localhost, which is the default location where the client sends the spans. Typically, the agent would be deployed as a sidecar in such a scenario:

https://github.com/jaegertracing/jaeger-kubernetes#deploying-the-agent-as-sidecar