Get details of All Class A operations in GCP Cloud Storage to Understand Cost

Question

I am running a large data transfer between buckets using data transfer service for cloud storage.

I want to find out exactly how many times I am invoking ClassA operations and ClassB operations to understand and analyse cost trends.

Would like to know if there is an easy way to achieve it programmatically or via console ?

Answer 1

Enable Audit Logs

Follow the link and Enable Audit Logging for buckets in your projects with

Admin Read
Data Read
Data Write

https://cloud.google.com/logging/docs/audit/configure-data-access#config-console

Cloud Logging

Most Class A Operations

Use this regex in cloud logging logs explorer to find out all classA operations invoked and select time frame as well

resource.type="gcs_bucket"
protoPayload.methodName=~"(storage.(buckets|objects|notifications|projects|.*AccessControls).(list|lockRetentionPolicy|compose|copy|rewrite|watchAll|delete|hmacKeys|insert|patch|update|setIamPolicy))"

Most Class B Operations

Use this regex in cloud logging logs explorer to find out all classB operations invoked and select time frame as well

resource.type="gcs_bucket"
protoPayload.methodName=~"(storage.(.*).(get|getIamPolicy|testIamPermissions|AccessControls\.list|))"

To filter further your data, you can include principal

resource.type="gcs_bucket"
protoPayload.authenticationInfo.principalEmail=“[email protected] "

Note : It will increase the cost due to cloud logging.