Jaeger and kubernetes with a public collector ingress

112 Views Asked by At

Scenario

I am using helm to facilitate creation of deployments and services for jaeger. Specifically https://github.com/jaegertracing/helm-charts

I have an application running on the same k8s that pushed traces to http://jaeger-collector.observability.svc.cluster.local:4317 and these work fine. (observability is the namespace where jaeger is installed)

Now i also have an external application which I want to forward traces to jaeger aswell.

Helm chart values below

...ommitted...
collector:
  enabled: true
  basePath: /
  service:
    otlp:
      grpc:
        name: otlp-grpc
        port: 4317
      http:
        name: otlp-http
        port: 4318
  ingress:
    enabled: true
    ingressClassName: nginx
    hosts:
      host: collector.example.com
...ommitted...

Ingress Problem and workaround

However there is a problem in the above helm chart, it seems to only let me create an ingress for port 14268 only. I have worked around that, and for testing purposes I have updated the generated yml file directly on k8s, and replaced the port as below

spec:
  ingressClassName: nginx
  rules:
    - host: collector.example.com
      http:
        paths:
          - path: /
            pathType: ImplementationSpecific
            backend:
              service:
                name: jaeger-collector
                port:
                  number: 4317 #was 14268

My expectations were that nginx (ingress controller) will be listetning on port 80 and 443 and will forward all requests to jaeger-collector.

Analysis

This still doesnt work. My collector service is working correctly 100% as applications running on k8's manage to push data. So the problem seems to be something with the ingress or something before that.

From what i can tell, the (modified) ingress is correct. My application code is .Net C#.

Application code

public static IServiceCollection AddAppTelemetry(this IServiceCollection services, IConfiguration configuration, IWebHostEnvironment webHostEnvironment)
{
    services.AddOpenTelemetry()
        .ConfigureResource(r => r.AddService(serviceName: webHostEnvironment.ApplicationName))
        .WithTracing(builder =>
        {
            builder
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddNpgsql()
            .AddRedisInstrumentation()
            .AddOtlpExporter();
        });

    services.Configure<OtlpExporterOptions>(configuration.GetSection("OpenTelemetry:otlp"));

    return services;

}

and my appsettings options

"OpenTelemetry": {
  "otlp": {
    "Endpoint": "http://collector.example.com:80"
  }
},

Possible issues and solution attempts

  1. I am running collector.example.com behind cloudflare proxy, I disabled this to no avail.
  2. Curl and telnet to collector.example.com works fine.
  3. Tried creating my own ingress instead of the helm one, no luck
  4. My own service that listens on port 80/443 and redirects to 4317

At this point I'm out of ideas, I only just learned Helm and jaeger since yesterday, and im quite fresh with k8s aswell. Any help is really appreciated.

0

There are 0 best solutions below