Google Cloud Run outbound static IP is 169.254.X.X instead of reserved one

Question

I created a Google Cloud Run revision with a Serverless VPC access connector to a VPC Network. The VPC Network has access to the internet via Cloud NAT, to allow the Cloud Run instance to have a static outbound ip address, as described in this tutorial: https://cloud.google.com/run/docs/configuring/static-outbound-ip. I followed the tutorial, and all was well, I got a static ip adress on egress traffic from the Cloud Run instance.

I used terraform to deploy all the resources, the code of which you can find below. The problem is this: After destroying the resources, I got following error:

ERROR: (gcloud.compute.networks.delete) Could not fetch resource:
 - The network resource 'projects/myproject/global/networks/webhook-network' is already being used by 'projects/myproject/global/networkInstances/v1823516883-618da3a7-bd4f-4524-...-...' 

(the dots contain more numbers, but as this seems to be some kind of uuid I prefer not to share the rest).

So I can't delete the first network. When I change the network's name and reapply, the apply succeeds, but the outbound static ip address of the egress is 169.254.X.X, which I find the following information about:

"When you see a 169.254.X.X, you definitely have a problem" ==> smells like trouble.

Any Googlers that can help me out? I think the steps to reproduce the 'corrupted' VPC network is to create a Serverless Access Connector with a connection to the VPC, reference it with a Cloud Run revision, and then delete the VPC network and the Serverless Access Connector before you delete the Cloud Run revision, but honestly not sure, I don't really have spare GCP projects laying around to test it out on.

This StackOverflow question did not help out: https://serverfault.com/questions/1016015/google-cloud-platform-find-resource-by-full-resource-name, and it's the only related one I can find.

Anyone have any ideas?

locals {
  region = "europe-west1"
}

resource "google_compute_network" "webhook_network" {
  name = "webhook-network-6"
  auto_create_subnetworks = false
}

resource "google_compute_subnetwork" "subnetwork" {
  depends_on = [
    google_compute_network.webhook_network
  ]
  name = "webhook-subnet-6"
  network = google_compute_network.webhook_network.self_link
  ip_cidr_range = "10.14.0.0/28"
  region = local.region
}

resource "google_compute_router" "router" {
  depends_on = [
    google_compute_subnetwork.subnetwork,
    google_compute_network.webhook_network
  ]
  name = "router6"
  region = google_compute_subnetwork.subnetwork.region
  network = google_compute_network.webhook_network.name
}

// I created the static IP address manually
//resource "google_compute_address" "static_address" {
//  name = "nat-static-ip-address"
//  region = local.region
//}

resource "google_compute_router_nat" "advanced-nat" {
  name = "natt"
  router = google_compute_router.router.name
  region = local.region
  nat_ip_allocate_option = "MANUAL_ONLY"
  nat_ips = [
    var.ip_name
  ]
  source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES"

}

data "google_project" "project" {

}


}
resource "google_vpc_access_connector" "access_connector" {
  depends_on = [
    google_compute_network.webhook_network,
    google_compute_subnetwork.subnetwork
  ]
  name = "stat-ip-conn-6"
  project = var.project_id
  region = local.region
  ip_cidr_range = "10.4.0.0/28"
  network = google_compute_network.webhook_network.name
}

Answer 1

Turns out I was working correctly, the way to test it was wrong. I was testing it using following Cloud Function:

def hello_world(request):
    request_json = request.get_json()
    ip = request.remote_addr # the culprit
    remote_port = request.environ.get('REMOTE_PORT')
    url = request.url
    host_url = request.host_url
    return {"ip": ip, "url": url, "port": remote_port, "host_url": host_url}

which returns the 169.254.X.X, but when I test against curlmyip.org, it is indeed the correct ip address.

But, that still does not solve the issue of not being able to delete the VPC network.