TPU quota exceeded on project xxxx on Google Cloud Console

Question

I am trying to initialize a TPU on Google Cloud Console on a particular project.

On typing ctpu up I am prompted with the following usual message for me to confirm if I want to initialize the TPU or not.

  Zone:                 us-central1-b
  GCP Project:          xxxx
  TensorFlow Version:   2.2
  VM:
      Machine Type:     n1-standard-2
      Disk Size:        250 GB
      Preemptible:      false
  Cloud TPU:
  Cloud TPU:
      Size:             v2-8
  Cloud TPU:
      Size:             v2-8
      Preemptible:      false
      Reserved:         false

On confirming with Y I am prompted with the message that

TPU quota exceeded on project xxxx.

On checking the Quotas page I see, there is 0 requests from cloud TPUs this month let alone reaching anywhere near the quota limit.

All of this with the fact that I have regularly used it for the previous two months, have a credit card associated with the account. Have paid my last month's bill where I had extensively created TPUs and used them on my project. Usually I used to work with preemptibled TPUs. However, I am neither able to create preemptible nor non-preemptible TPUs.

I have tried with all the available regions. I have tried with all the available TPU versions. I have tried both preemptible and non-premptible type for all available TPU versions.

Is there anyway I can regain control back to create TPUs on my account or can someone shed some light into what exactly happened?

Answer 1

When you are in the Quotas view, add a filter set to "TPU v2 cores per project per zone". Find the row with us-central1-b (or whatever zone you are currently targeting). What is the "Limit" value?

If it's 0, try another zone and/or tpu configuration that has a non-zero limit.

If it's > 0, you might have some inactive tpus hanging around that are consuming the quota - you can check this with gcloud compute tpus list --zone=us-central1-b.