I am trying to deploy a native app as a lambda function and set it up to submit traces to AWS X-Ray. After couple days I'm still not sure if it's possible.
I've added a sample app that I deploy to AWS: https://github.com/DSkoufis/quarkus-otel-lambda
The ideal scenario is to create spans using OpenTelemetry's API with @WithSpan annotation. This works fine in a normal app but in the lambda it's unable to capture the parent span - so I assume it has something to do with the propagation of the headers.
For the lambda I've used the quarkus-amazon-lambda extension where it also mentions AWS X-Ray in the docs: https://quarkus.io/guides/aws-lambda#tracing-with-aws-xray-and-graalvm
I'm not using manage.sh to deploy the lambda but Terraform but it's essentially a copy of what the manage.sh does, i.e:
https://github.com/DSkoufis/quarkus-otel-lambda/blob/main/terraform/lambda.tf#L23
tracing_config {
mode = "Active"
}
If I test this lambda through AWS Console it submits traces but not for the manual spans and also the trace context is not propagated correctly in order for the traces to appear as parent of the manually created ones, i.e https://github.com/DSkoufis/quarkus-otel-lambda/blob/main/src/main/java/org/acme/opentelemetry/lambda/LambdaService.java#L20
START RequestId: 09603df8-91a8-4098-8a37-c6d4e15d03fc Version: $LATEST
15:41:04 INFO traceId=e1cb9f4b723b5f6c0d33dbdc26af86d2, parentId=, spanId=b2671df3374f3863, sampled=true [or.ac.op.la.LambdaService] (Lambda Thread (NORMAL)) Request ID: value1
15:41:04 INFO traceId=e1cb9f4b723b5f6c0d33dbdc26af86d2, parentId=b2671df3374f3863, spanId=7fcf32d3009c543d, sampled=true [or.ac.op.la.LambdaService] (Lambda Thread (NORMAL)) Here
END RequestId: 09603df8-91a8-4098-8a37-c6d4e15d03fc
I've also tried to use the ID Generator example from here: https://quarkus.io/guides/opentelemetry#id-generator but adding the opentelemetry-aws-xray doesn't compile anymore to native
<dependency>
<groupId>io.opentelemetry.contrib</groupId>
<artifactId>opentelemetry-aws-xray</artifactId>
<version>1.32.0</version>
</dependency>
(or event using the IdGenerator suggested here: https://github.com/quarkusio/quarkus/discussions/33154 also didn't work)
I've also tried to add a layer using the AWS Distro for OpenTelemetry Collector but still nothing (commented out in terraform for now):
layers = [
"arn:aws:lambda:${var.aws_region}:${var.account_no}:layer:aws-otel-java-agent-${var.lambda_architecture}-ver-1-32-0:1"
]