gcloud - BrokerCell cloud-run-events/default is not ready

Question

I am trying to use google cloud for my pubsub event driven application. Currently, I am setting up Cloud Run for Anthos following the below tutorials

• https://codelabs.developers.google.com/codelabs/cloud-run-events-anthos#7
• https://cloud.google.com/anthos/run/archive/docs/events/cluster-configuration

I have created the GKE clusters. It is successful and is up and running.

However, I am getting the below error when I try to create event broker.

$ gcloud beta events brokers create default --namespace default

X Creating Broker... BrokerCell cloud-run-events/default is not ready                                                                                        
  - Creating Broker...                                                                                                                                       
Failed.                                                                                                                                                      
ERROR: gcloud crashed (TransportError): HTTPSConnectionPool(host='oauth2.googleapis.com', port=443): Max retries exceeded with url: /token (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))

gcloud's default CA certificates failed to verify your connection, which can happen if you are behind a proxy or firewall.
To use a custom CA certificates file, please run the following command:
  gcloud config set core/custom_ca_certs_file /path/to/ca_certs

However, When I rerun the command, it shows broker already exists

$ gcloud beta events brokers create default --namespace default

ERROR: (gcloud.beta.events.brokers.create) Broker [default] already exists.

Checking the status of broker, it shows BrokerCellNotReady

$ kubectl get broker -n default

NAME     URL                                                        AGE   READY     REASON

default  http://default-brokercell-ingress.cloud-run-events.svc.cluster.local/default/default   39m   Unknown   BrokerCellNotReady

And I am getting status pending for default-brokercell-fanout pod.

$ kubectl get pods -n cloud-run-events

NAME                                          READY   STATUS    RESTARTS   AGE
controller-648c495796-b5ccb                   1/1     Running   0          105m
default-brokercell-fanout-855494bb9b-2c7zv    0/1     Pending   0          100m
default-brokercell-ingress-5f8cdc6467-wwq42   1/1     Running   0          100m
default-brokercell-retry-6f4f9696d6-tg898     1/1     Running   0          100m
webhook-85f7bc69b4-qrpck                      1/1     Running   0          109m

I couldn't find any discussion related to this error. Please give me some ideas to resolve this issue.

Answer 1

I encountered the same issue. The reason might be the given cluster setup does not have enough CPU resources.

You can check it by

kubectl describe pod/default-brokercell-retry-6f4f9696d6-tg898 -n cloud-run-events

If the output is then that's the reason.

After knowing the root cause, you can fix it in various ways, e.g., enable auto-scaling in your node pool.