Insufficient CPU quota when running data quality task in GCP dataplex

Question

I try to follow the below guide from GCP to create a data quality task. https://cloud.google.com/dataplex/docs/check-data-quality?&_ga=2.139058355.-143783534.1669209779#before_you_begin when I try to run the task I get the error

• As per my understanding Dataplex is calling dataproc internally for running the spark jobs due to which we get the error message. But we dont have the option to adjust the number of CPUs that can be used.
• I tried to increase the CPU quota but GCP does not allow me( My account is enabled for billing for a long time) Any suggestion to correct this error will be appreciated.

Answer 1

As mentioned by @guillaume, there is set a limit on the number of CPUs you can use for a region (and all regions) in GCP. It is different for individuals and organizations. You can check this under IAM -> Quotas-> Compute Engine API CPU quota.

Solutions to your error:

• You can request a quota increase on the number of CPUs for a region/all regions for your project.

• Try changing the configuration of your Dataproc Cluster by decreasing the number of secondary workers. Reference

Answer 2

It won't help you, but even if you have activated the billing, as personal user you have a very limited number of allowed CPU (here my CPU quotas for all region; and I'm a GDE! Check it in your project.)

And here the quotas with my corporate sandbox project

If you have the opportunity to test it in a corporate account, or if you have a Google Cloud contact, you could ask for a quota increase.