Google cloud storage object - Service account does not have storage object get access

1.8k Views Asked by At

I have a service account for my GCP project that uses the Vault and Google Cloud Storage APIs. I have ensured that both of these APIs are enabled. I have also ensured my service account has the correct roles ( which have the permissions inherited ? ) for modifying storage objects ( I am downloading Exports from vault to my local server for archiving, which requires the get access / write permissions ) Storage Admin and Storage Object Admin. Do I have to generate a new service account Json Key if I modified the roles/permissions of the SA account after creation in order for the credentials to be updated in the API calls from the client or will they be updated with the same original SA key I made and continue to reference in the method api calls to get cloud storage objects ? Or am I simply not using SA credentials correctly for these APIs ?

IAM service account showing roles/permissions

Cloud Storage API showing enabled and service account credentials

For clarity, this is what an export data object contains:

{'exports': [{'id': 'v3_MAIL_241a6eba-d2b3-45b1-83fd-fd343b3cf750', 'matterId': '4bbc1e6e-85c2-467e-b967-1c0e83ed90e4', 'name': 'test export', 'requester': {'email': '[email protected]', 'displayName': 'Alexi Richardson'}, 'query': {'corpus': 'MAIL', 'dataScope': 'ALL_DATA', 'searchMethod': 'ACCOUNT', 'accountInfo': {'emails': ['[email protected]']}, 'mailOptions': {}, 'timeZone': 'America/Chicago', 'method': 'ACCOUNT'}, 'exportOptions': {'mailOptions': {'exportFormat': 'MBOX', 'showConfidentialModeContent': True, 'useNewExport': True}, 'region': 'US'}, 'createTime': '2022-05-05T21:17:37.695Z', 'status': 'COMPLETED', 'stats': {'exportedArtifactCount': '20856', 'totalArtifactCount': '20856', 'sizeInBytes': '1133035820'}, 'cloudStorageSink': {'files': [{'bucketName': '9b750b66-7789-466d-9ad4-c41c78529347', 'objectName': '50284674:4bbc1e6e-85c2-467e-b967-1c0e83ed90e4:v3_MAIL_241a6eba-d2b3-45b1-83fd-fd343b3cf750/a830db3a-8317-4fba-ad20-9d70415f67d6', 'size': '1126473769', 'md5Hash': 'b7c144cd03255b217b5ab6dd2a6c37c2'}, {'bucketName': '9b750b66-7789-466d-9ad4-c41c78529347', 'objectName': '50284674:4bbc1e6e-85c2-467e-b967-1c0e83ed90e4:v3_MAIL_241a6eba-d2b3-45b1-83fd-fd343b3cf750/686a456b-d796-439a-8ef3-beac4e071826', 'size': '6561479', 'md5Hash': '63cd3eac2c7cb41699c4e6b908cff72a'}, {'bucketName': '9b750b66-7789-466d-9ad4-c41c78529347', 'objectName': '50284674:4bbc1e6e-85c2-467e-b967-1c0e83ed90e4:v3_MAIL_241a6eba-d2b3-45b1-83fd-fd343b3cf750/5cc3e164-9758-4d39-9f71-2c3d0ad8555f', 'size': '442', 'md5Hash': '75e7e6081ab433267ec1513c39970d3f'}, {'bucketName': '9b750b66-7789-466d-9ad4-c41c78529347', 'objectName': '50284674:4bbc1e6e-85c2-467e-b967-1c0e83ed90e4:v3_MAIL_241a6eba-d2b3-45b1-83fd-fd343b3cf750/343f5811-83a7-4165-814f-18a8cbb1649e', 'size': '130', 'md5Hash': '12746dd0443dd4fc1a2968b5866b2d6b'}]}}]}

I am invoking the SA key credentials explicitly in the function like this at the storage_client line:

#METHOD 1 for downloading exports
def download_exports_primary(service, matter_id):
 #""" Google Cloud storage service is authenticated by running `gcloud auth application-default login`
#    and expects a billing enabled project in ENV variable `GOOGLE_CLOUD_PROJECT` """

    storage_client = storage.Client.from_service_account_json('/opt/gws/creds/gws-vault-data-export-ops-bfd51297e810.json')

    for export in service.matters().exports().list(matterId=matter_id).execute()['exports']:
      if 'cloudStorageSink' in export:
        directory = export['name']
        if not os.path.exists(directory):
          os.makedirs(directory)
        print('\n')
        print(export['id'])
        for sinkFile in export['cloudStorageSink']['files']:
            filename = '%s/%s' % (directory, sinkFile['objectName'].split('/')[-1])
            objectURI = 'gs://%s/%s' % (sinkFile['bucketName'],sinkFile['objectName'])
            print('\n')
            print('Pulling export download....')
            print('\n')
            print('get %s to %s' % (objectURI, filename))
            print('\n')
            #storage_client.download_blob_to_file(objectURI, open(filename, 'wb+'))
            try:
                with open(filename, 'wb') as outfile:
                    storage_client.download_blob_to_file(objectURI, outfile)
            except Exception as e:
                print(e)
            else:
                print('No exception')


However, I keep getting this error:

403 GET https://storage.googleapis.com/download/storage/v1/b/9b750b66-7789-466d-9ad4-c41c78529347/o/a5b3ae10-2e47-4a03-9b4d-4856c9f13356%2Fexportly-00b683d9-807c-41f8-b9fd-c1e82281ba9d%2Fj.howell%40striveworks.usGMAIL_EXPORT-1.zip?alt=media: [email protected] does not have storage.objects.get access to the Google Cloud Storage object.: ('Request failed with status code', 403, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)
0

There are 0 best solutions below