Unable to get Istio Gateway Logs in stackdriver

Question

I have deployed Istio on private GKE with WLI enabled. Istio is being installed with Istioctl and istio-operator. I want to see if my outbound requests are stuck anywhere and I am unable to see so as I am getting the following error:

CreateTimeSeries request failed (1 RPCs, 16 views, 20 timeseries): PERMISSION_DENIED: Permission monitoring.timeSeries.create denied (or the resource may not exist). I could see the serviceaccount attached to the gateway is istio-egressgateway-service-account which is not explicitly created by me. I think this is created by ISTIO . So I would like like to understand how can I fix this. I am a bit worried to attach this service account to GCP IAM service Account as it was managed by Istio and I don’t want to disturb that.

Is it because of Work Load identity(WLI) or anything else? And How can I fix this. Any ideas and help would be very much appreciated. GKE Version: 1.17.9-gke.1504 Istio Version 1.7.x

Answer 1

for anyone who is searching and viewing this page:

refer: GKE workload identity

export GCP_PROJECT=my-project
export GCP_SA=gke-prometheus
export K8S_SA=prometheus
export K8S_NS=prometheus

gcloud iam service-accounts create ${GCP_SA} --display-name=${GCP_SA}

gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --member "serviceAccount:${GCP_PROJECT}.svc.id.goog[${K8S_NS}/${K8S_SA}]" \
  ${GCP_SA}@${GCP_PROJECT}.iam.gserviceaccount.com

gcloud projects add-iam-policy-binding ${GCP_PROJECT} \
  --member "serviceAccount:${GCP_SA}@${GCP_PROJECT}.iam.gserviceaccount.com" \
  --role roles/monitoring.metricWriter

gcloud projects add-iam-policy-binding ${GCP_PROJECT} \
  --member "serviceAccount:${GCP_SA}@${GCP_PROJECT}.iam.gserviceaccount.com" \
  --role roles/monitoring.viewer


gcloud projects add-iam-policy-binding ${GCP_PROJECT} \
  --member "serviceAccount:${GCP_SA}@${GCP_PROJECT}.iam.gserviceaccount.com" \
  --role roles/logging.logWriter


gcloud projects add-iam-policy-binding ${GCP_PROJECT} \
  --member "serviceAccount:${GCP_SA}@${GCP_PROJECT}.iam.gserviceaccount.com" \
  --role roles/stackdriver.resourceMetadata.writer


kubectl annotate serviceaccount ${K8S_SA} \
  iam.gke.io/gcp-service-account="${GCP_SA}@${GCP_PROJECT}.iam.gserviceaccount.com" \
  -n ${K8S_NS}