which permission is missing to initialize spanner client?

Question

Tried to create spanner client in gke pods, but got:

File "/usr/local/lib/python3.7/site-packages/google/cloud/spanner_v1/database.py", line 519, in run_in_transaction
     with SessionCheckout(self._pool) as session:
   File "/usr/local/lib/python3.7/site-packages/google/cloud/spanner_v1/pool.py", line 536, in __enter__
     self._session = self._pool.get(**self._kwargs)
   File "/usr/local/lib/python3.7/site-packages/google/cloud/spanner_v1/pool.py", line 273, in get
     session.create()
   File "/usr/local/lib/python3.7/site-packages/google/cloud/spanner_v1/session.py", line 117, in create
     session_pb = api.create_session(self._database.name, metadata=metadata, **kw)
   File "/usr/local/lib/python3.7/site-packages/google/cloud/spanner_v1/gapic/spanner_client.py", line 307, in create_session
     request, retry=retry, timeout=timeout, metadata=metadata
   File "/usr/local/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__
     return wrapped_func(*args, **kwargs)
   File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
     on_error=on_error,
   File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 206, in retry_target
     last_exc,
   File "<string>", line 3, in raise_from

google.api_core.exceptions.RetryError: Deadline of 3600.0s exceeded while calling functools.partial(<function _wrap_unary_errors.<locals>.error_remapped_callable at 0x7f8bff413ef0>,
database: "projects/myproj-1501/instances/tfgen-spanid-2020585/databases/spanner-stage,
metadata=[('google-cloud-resource-prefix', 'projects/myproj-1501/instances/tfgen-spanid-2020585/databases/spanner-stage'),
('x-goog-request-params',
'database=projects/myproj-1501/instances/tfgen-spanid-2020585/databases/spanner-stage'),
 ('x-goog-api-client', 'gl-python/3.7.9 grpc/1.32.0 gax/1.22.2 gapic/1.17.1 gccl/1.17.1')]),
 last exception: 503 Getting metadata from plugin failed with error: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/[email protected]/token from the Google Compute Enginemetadata service.
  Status: 403 Response:\nb'Unable to generate access token; IAM returned 403 Forbidden: The caller does not have permission\\nThis error could be caused by a missing IAM policy binding on the target IAM service account.
  \\nFor more information, refer to the Workload Identity documentation:\\n\\thttps://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#creating_a_relationship_between_ksas_and_gsas\\n\\n'", <google.auth.transport.requests._Response object at 0x7f8bfcb33810>)

Any idea how to figure out which permission is missing? which service account needs this permission?

Thanks

Answer 1

Under this article, step 2 says how to grant a role and points to one of these roles. I suspect you would need one of these two roles:

roles/spanner.admin

roles/spanner.databaseAdmin

Too many steps to list here and it depends on the account, but Step1 in the first article shows you how to identify the proper service account. Note that GKE users GCE, so the service account may simply look like a regular "Compute Engine" service account.

Answer 2

The error message suggests that there may be a missing IAM policy binding on the target IAM service account '[email protected]'. Are you able to follow the Workload Identity documentation?

Additionally, you need to grant your service account permission to access your Cloud Spanner databases. You may follow the instruction here.