I'm trying to create a dataproc cluster on GKE, through the gcp console using the following command
gcloud dataproc clusters gke create dp-gke-cluster --region=europe-west1 --gke-cluster=gke-cluster --spark-engine-version=latest --pools="name=dp-gke-pool,roles=default" --setup-workload-identity
as written in this guide But I keep getting this error message and I can't wrap my head around it.
ERROR: (gcloud.dataproc.clusters.gke.create) Operation [projects/test-project/regions/europe-west1/operations/824ad550-9e03-363f-94b4-cfc1c942379e] failed: Error creating 'projects/test-project/locations/europe-west1/clusters/gke-cluster/nodePools/dp-gke-pool': Request contains an invalid argument..
The dataproc cluster is created but it has no nodes in that, because the nodepool is never created.
has anyone faced the same problem? or knew where is the problem? thx in advance.
I've created different clusters with different number of nodes, but obviously it doesn't change anything since dataproc should create a different nodepool on that cluster. But that nodepool is never created.
After replicating the issue and trying extensively to find the solution, it seems that there is an issue in either the documentation or the product. So, for further support you can raise a support ticket with GCP support team or raise a ticket in the public issue tracker.