GKE node upgrade stuck "Error 400: Cluster is running incompatible operation operation"

286 Views Asked by At

I have a GKE cluster which was trying to upgrade it's nodes. But seems the operation got stuck and I can see the below error for the last 2 days.

enter image description here

When I try to run any command using the CLI it will give me the below error,

ERROR: (gcloud.container.clusters.update) FAILED_PRECONDITION: Cluster is running incompatible operation operation-1700226020173-210a422c-f4b9-41f0-9e21-f9bee66c5535.'@type': type.googleapis.com/google.rpc.RequestInfo requestId: '0x290efa3770b767d8'

gcloud container operations list command giving an output telling the operation still aborting.

023-11-17T12:53:32.0282925Z    2023-11-17T12:53:35.327130441Z
operation-1700226020173-210a422c-f4b9-41f0-9e21-f9bee66c5535  UPGRADE_NODES       
europe-west4  example-cluster-node-pool-1                                                                                                                                                                                                                                                                                                                                                                                                                         
ABORTING  2023-11-17T13:00:20.173869352Z

Since this, I cannot edit the cluster in the console as well. If i try to redeploy using terraform it will timeout telling the above : googleapi: Error 400: Cluster is running incompatible operation operation error. 

Appreciate your feedback to resolve this.

0

There are 0 best solutions below