Exception thrown when exporting TimeSeries for projectName=XXX

Question

I am trying to enable client/server side metrics for one of my springboot application that is deployed in GCP cloud run, and write data to Bigtable.

before enabling the client/server side metrics the app was working fine and no problem in processing request. since we enable the metric writer I am continuously seeing different errors.

"com.google.api.gax.rpc.UnknownException: io.grpc.StatusRuntimeException: UNKNOWN: Authentication backend unknown error.
    at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:119)
    at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:41)
    at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:86)
    at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66)
    at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
    at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:67)
    at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1132)
    at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
    at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1270)
    at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1038)
    at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:808)
    at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:574)
    at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:544)
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
    at com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:535)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:563)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:744)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)
    Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed
        at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57)
        at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112)
        at com.google.cloud.monitoring.v3.MetricServiceClient.createServiceTimeSeries(MetricServiceClient.java:1876)
        at com.google.cloud.bigtable.stats.BigtableCreateTimeSeriesExporter.export(BigtableCreateTimeSeriesExporter.java:80)
        at com.google.bigtable.veneer.repackaged.io.opencensus.exporter.metrics.util.MetricReader.readAndExport(MetricReader.java:167)
        at com.google.bigtable.veneer.repackaged.io.opencensus.exporter.metrics.util.IntervalMetricReader$Worker.readAndExport(IntervalMetricReader.java:177)
        at com.google.bigtable.veneer.repackaged.io.opencensus.exporter.metrics.util.IntervalMetricReader$Worker.run(IntervalMetricReader.java:170)
        ... 1 common frames omitted
Caused by: io.grpc.StatusRuntimeException: UNKNOWN: Authentication backend unknown error.
    at io.grpc.Status.asRuntimeException(Status.java:539)
    ... 14 common frames omitted

the new code I've added is

BigtableDataSettings.enableBuiltinMetrics();

I've checked the IAM policies and everything looks good in place. not sure what I am missing here, any help would be greatly appreciated. thanks!

Answer 1

The authentication backend unknown should be a transient error. Does it happen all the time or is it reproducible at least? Are you seeing other errors thrown from export time series?

If you have any special set up for Bigtable client, you should set up credentials for your metrics with BigtableDataSettings.enableBuiltinMetrics(Credentials credentials);. Make sure you've followed all the steps for Bigtable authentication: https://cloud.google.com/bigtable/docs/authentication and you've enabled cloud monitoring API https://cloud.google.com/bigtable/docs/client-side-metrics-setup#enable-the-cloud-monitoring-api.