SpringBoot Sleuth/Zipkin Tracing: Issue adding a client, for Zipkin Server

Question

Small question regarding a Java 11 + SpringBoot 2.6.7 web application, in the context of Zipkin please.

I have a very simple micro services architecture SpringBoot repo that can be found here: https://github.com/dogukankrtlz/springboot-sleuth-zipkin, please feel free to clone.

Here are the relevant code for each micro service, it is very straightforward, I omitted the POJOs etc...

Main service, called customer service, is calling to call two child services:

    @RequestMapping(value="/customer/{cid}", method=RequestMethod.GET)
    public CustomerDetails getCustomer(@PathVariable String cid) {

        CustomerDetails customer = new CustomerDetails();

        ContactDetails svc1 = wc.get()
                .uri("http://localhost:8081/customer/" + cid + "/contactdetails")
                .retrieve()
                .bodyToMono(ContactDetails.class)
                .block();

        VehicleDetails svc2 = wc.get()
                .uri("http://localhost:8082/customer/" + cid + "/vehicledetails")
                .retrieve()
                .bodyToMono(VehicleDetails.class)
                .block();

        customer.setContactId(cid);
        customer.setContactName(svc1.getContactName());
        customer.setPostalCode(svc1.getPostalCode());
        customer.setLicensePlate(svc2.getLicensePlate());
        customer.setCarType(svc2.getCarType());

        return customer;
    }

Calling service A

 @RequestMapping(value="/customer/{cid}/contactdetails", method= RequestMethod.GET)
    public ContactDetails getCustomerContactDetails(@PathVariable String cid) throws InterruptedException {
        Span dbSpan = this.tracer.nextSpan().name("DBLookup");
        try (Tracer.SpanInScope ws = this.tracer.withSpan(dbSpan.start())) {

            dbSpan.tag("call", "sql-database");

            Random r = new Random();
            int multiplier = r.nextInt(5) * 1000;
            Thread.sleep(multiplier);

            dbSpan.event("db lookup complete");
        }
        finally {
            dbSpan.end();
        }

        return details.stream().filter(detail -> cid.equals(detail.getContactId())).findAny().orElse(null);

    }

Then calling service B

 @RequestMapping(value="/customer/{cid}/vehicledetails", method=RequestMethod.GET)
    public VehicleDetails getCustomerVehicleDetails(@PathVariable String cid) {

        return details.stream().filter(detail -> cid.equals(detail.getCustomerId())).findAny().orElse(null);
    }

Once I make a request, I can see in the logs the traces, things are working, happy:

DEBUG [customerservice,25ef30dd73fbb6d8,25ef30dd73fbb6d8] 26578 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed 200 OK
DEBUG [dataservice1,25ef30dd73fbb6d8,65be0bf2df917b35] 26576 --- [nio-8081-exec-1] o.s.web.servlet.DispatcherServlet        : Completed 200 OK
DEBUG [dataservice2,25ef30dd73fbb6d8,a490771db3881b76] 26570 --- [nio-8082-exec-1] o.s.web.servlet.DispatcherServlet        : Completed 200 OK

Please note, I can see all trace IDs fine: 25ef30dd73fbb6d8,25ef30dd73fbb6d8 on the Main service, 25ef30dd73fbb6d8,65be0bf2df917b35 and 25ef30dd73fbb6d8,a490771db3881b76 on child services.

Most important part, I can even see the traces in Zipkin Server, they are all here, beautiful, will all relevant information: (see screenshot)

Things are working quite well, so far. Now, I would like to add a client. This is where things start to go south.

I add this client, also very straightforward:

 public static void main(String[] args) {
        final OkHttpSender        sender    = OkHttpSender.newBuilder().endpoint("http://localhost:9411/api/v2/spans").build();
        final AsyncReporter<Span> reporter  = AsyncReporter.create(sender);
        final Tracing             tracing   = Tracing.newBuilder().localServiceName("ClientService").addSpanHandler(ZipkinSpanHandler.create(reporter)).build();
        final BraveTracer               tracer    = BraveTracer.create(tracing);
        final BraveSpan                 parent    = tracer.buildSpan("client start API").start();
        final WebClient           webClient = WebClient.create().mutate().defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE, HttpHeaders.ACCEPT, MediaType.APPLICATION_JSON_VALUE).clientConnector(new ReactorClientHttpConnector(HttpClient.create().responseTimeout(Duration.ofSeconds(10)).wiretap(true).protocol(HttpProtocol.HTTP11))).build();
        final String              response  = webClient.get().uri("http://localhost:8080/customer/500").header("X-B3-TraceId", parent.unwrap().context().traceIdString()).header("X-B3-SpanId", parent.unwrap().context().spanIdString()).retrieve().bodyToMono(String.class).onErrorReturn("ERROR").block();
        parent.finish();
        reporter.flush();
        System.out.println("->" + response);
    }

I do see logs from both client and server:

client: X-B3-TraceId: d96d2354f60b306a..X-B3-SpanId: d96d2354f60b306a
server:
DEBUG [customerservice,d96d2354f60b306a,d96d2354f60b306a]
DEBUG [dataservice1,d96d2354f60b306a,f61a595dfd045e17]
DEBUG [dataservice2,d96d2354f60b306a,df125c7d44c5143c]

I go back into Zipkin with hope, and this is all I see:

What I would expect to see, is definitely something more like this (photoshop).

May I ask what went wrong here?

How to achieve a result where I can see the complete trace, from client all the way to server, all in one place in Zipkin?

Thank you

Answer 1

You can't create those objects manually (those objects == tracer, webclient, tracing etc.). You have to get them from the application context as beans. Then all the inbuilt instrumentations will be applied.

UPDATE:

Since it seems you're not using Spring Cloud Sleuth in all projects, you can still achieve sth similar by doing the instrumentation manually. That means that you would have to add the tracing exchange functions to web client etc. In general all the instrumentations that Sleuth does automatically behnd the scenes you would have to do manually.