how can I make histogram latency buckets available to awsemf exporter (like grpc_server_handling_seconds_bucket)

152 Views Asked by At

I'm using open telemetry collector.

I have a prometheus scraper receiver and a awsemf exporter.

I can't figure out how to generate entries for the the latency buckets that I have defined in my prometheus configuration.

I can see debug information that shows that those data points are being generated.

ExplicitBounds #0: 0.050000
ExplicitBounds #1: 0.120000

Those are the buckets I have defined via configuration of my prometheus middleware.

...
import "github.com/prometheus/client_golang/prometheus"
...
        var opts *prometheus.HistogramOpts
...
    bucket1 := (50 * time.Millisecond).Seconds()
    bucket2 := (120 * time.Millisecond).Seconds()
    opts.Buckets = []float64{bucket1, bucket2}
...
}

But it is not translating to log entries in the awsemf format. (..."_aws": {}...etc)

I have a prometheus exporter that I use to see the metrics for testing in my machine, this exporter I think is not doing any mogrification AFAIK.

when I search an specific metric in the prometheus UI like:

grpc_server_handling_seconds_bucket

I see many results like

grpc_server_handling_seconds_bucket{... my labels ..., le="0.05", ... more of my labels ...}

for le, +Inf , 0.05 and 0.12 which are my latency buckets, why is awsemf not picking up that? (it is only picking up grpc_server_handling_seconds (no _bucket))

below is a snippet from open telemetry collector logs:

Metric #34
Descriptor:
     -> Name: grpc_server_handling_seconds
     -> Description: Histogram of response latency (seconds) of gRPC that had been application-level handled by the server.
     -> Unit: 
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
Data point attributes:
     -> custom_label1: Str(custom_label1)
     -> custom_label2: Str(custom_label2)
     -> grpc_method: Str(GRPCMETHOD)
     -> grpc_service: Str(GRPCSERVICE)
     -> grpc_type: Str(unary)
StartTimestamp: 2023-10-18 05:45:19.598 +0000 UTC
Timestamp: 2023-10-18 05:47:19.601 +0000 UTC
Count: 0
Sum: 0.000000
ExplicitBounds #0: 0.050000
ExplicitBounds #1: 0.120000
Buckets #0, Count: 0
Buckets #1, Count: 0
Buckets #2, Count: 0
HistogramDataPoints #1
Data point attributes:
     -> custom_label1: Str(custom_label1)
     -> custom_label2: Str(custom_label2)
     -> grpc_method: Str(GRPCMETHOD)
     -> grpc_service: Str(GRPCSERVICE)
     -> grpc_type: Str(unary)
StartTimestamp: 2023-10-18 05:45:19.598 +0000 UTC
Timestamp: 2023-10-18 05:47:19.601 +0000 UTC
Count: 0
Sum: 0.000000
ExplicitBounds #0: 0.050000
ExplicitBounds #1: 0.120000
Buckets #0, Count: 0
Buckets #1, Count: 0
Buckets #2, Count: 0

... many more datapoints and many more metrics ...

I tried adding dimensions that include le I've also tried detailed_metrics: true in the open telemetry config section for awsemf.

0

There are 0 best solutions below