Sending Traces to Grafana Tempo

Question

I've been trying to get into this for about 2 days. I have a .Net6.0 web application that needs to send tracing to a Grafana Tempo instance. Already configured is Jaeger for tracing, which is to be replaced. Unfortunately nothing I am trying is working and hope you guys can help me.

1. Researched how to approach this.

.Net has libraries that work with OpenTelemetry. However, only OpenTracing is necessary for this step. Tempo should also work directly with Jaeger, because jaeger already uses OpenTracing. 2. Jaeger already works with OpenTracing, so I tried to replace only the endpoint (http;//.....). Unsuccessfully

services.AddOpenTracing();
// Adds the Jaeger Tracer.
services.AddSingleton<ITracer>(sp =>
{
    var loggerFactory = sp.GetRequiredService<ILoggerFactory>();
    var reporter = new RemoteReporter.Builder()
        .WithLoggerFactory(loggerFactory)
        .WithSender(new HttpSender("http://localhost:9411"))
        .Build();
    var tracer = new Tracer.Builder(_serviceName)
        // The constant sampler reports every span.
        .WithSampler(new ConstSampler(true))
        // LoggingReporter prints every reported span to the logging framework.
        .WithReporter(reporter)
        .Build();
    return tracer;
});

services.Configure<HttpHandlerDiagnosticOptions>(options =>
    options.OperationNameResolver =
        request => $"{request.Method.Method}: {request?.RequestUri?.AbsoluteUri}");

using var scope = Tracer.BuildSpan(ControllerContext.ActionDescriptor.DisplayName)
    .StartActive(true);

Console.WriteLine(scope.Span.Context.TraceId);

scope.Span.Log("Requested all observation metadata");

3. Try to get it done with OpenTelemetry. I noticed that many NuGets are only available in the preview version.In addition, the documentation quickly shows that an OTelCollector is needed, but it is very vague about how to create one and connect it to Tempo. The collector was omitted here. It was to be expected that this would not work, but you can try it.

var serviceName = "Grafana.Tempo.POC.API";
var serviceVersion = "1.0.0";

// Configure important OpenTelemetry settings, the console exporter, and instrumentation library
builder.Services.AddOpenTelemetryTracing(
    tracerProviderBuilder =>
    {
        tracerProviderBuilder
            .AddConsoleExporter()
            .AddSource(serviceName)
            .SetResourceBuilder(
                ResourceBuilder.CreateDefault()
                    .AddService(serviceName: serviceName, serviceVersion: serviceVersion)
            )
            .AddHttpClientInstrumentation()
            .AddAspNetCoreInstrumentation()
            .AddSqlClientInstrumentation()
            .AddOtlpExporter(
                options =>
                {
                    options.Endpoint =
                        new Uri("http://localhost:9411");
                }
            );
    }
);

4. Then I read the Grafana documentation again and tested the Curl example. That was the first thing that arrived as a trace. However, this does not bring much, because I have to send this from the code. Yes I could manually drive the API in this form, but it is meaningless if I have to implement a tracing myself.

curl -X POST http://localhost:9411 -H 'Content-Type: application/json' -d '[{
 "id": "1234",
 "traceId": "0123456789abcdef",
 "timestamp": 1608239395286533,
 "duration": 100000,
 "name": "span from bash!",
 "tags": {
    "http.method": "GET",
    "http.path": "/api"
  },
  "localEndpoint": {
    "serviceName": "shell script"
  }
}]'

The Questions

• What would be the most useful and simplest way to get the tracing from ASPCore app to Grafana Tempo?
• Is it even possible to use Jaeger and simply redirect? Is it possible to use other configuration options?
• Or is it necessary to use the OpenTelemetry Collector to get everything running?

PS: I'm working with Win11, WSL2, Docker Desktop and .Net6.0 for testing. Later I want to run this in Kubernetes.

Answer 1

I know I'm late to the party but figured I'll share this video: https://www.youtube.com/watch?v=qhjVG8LKFgY

And a repository with a working sample of ASP.NET Core, Grafana, Tempo and a collector: https://github.com/bangonkali/aspnetcore-open-telemetry-grafana-tempo