I have this setup:
- An Exernal Global Load Balancer
- 1 Backend Services, with these backends:
- Backend 1 in us-central1, a cloud run serverless neg
- Backend 2 in us-east1, a cloud run serverless neg
- Backend 3 in me-central1, a cloud run serverless neg
- Routing rules are the default(simple path and host rule)
- No session affinity
- Balancing Policy: Least Request
- No CDN
Now being in Jordan, when I send a requst, I see it going to us-east1 service, despite me-central1 being the closest to me. And I don't see any errors or bad metrics on console.
So why is this happening?
The traffic should send the request to the closest region when multiple regions are used, but here in your case it is directed to the other closest region. This might be due to the region reaching the capacity or the backend is unhealthy and traffic overflowing to another closest region.
The external Application Load Balancer uses a special load balancing algorithm called Waterfall by Region.This algorithm determines the optimal backend for servicing the request by taking into account the proximity of the instances to the users, the incoming load as well as the available capacity of backends in each zone and region. Finally, worldwide load and capacity is also taken into account. The external Application Load Balancer distributes traffic based on available instances.
Traffic overflow to other regions
If an entire region reaches capacity as determined by the serving capacity set in the backend services, the algorithm is triggered, and traffic overflows to the closest region that has available capacity. As each region reaches capacity, traffic spills to the next closest region, and so on. A region's proximity to the user is defined by network round-trip time from the GFE to the instance backends.
Cross-regional overflow due to unhealthy backends
If health checks discover that more than half of the backends in a region are unhealthy, the GFEs preemptively overflow some traffic to the next closest region. This happens in order to avoid traffic completely failing as the region becomes unhealthy. This overflow occurs even if the remaining capacity in the region with the unhealthy backends is sufficient.
Also there are some limitations associated with configuring internet NEGs as backends. Connections to internet NEG endpoints use the internet while keeping your traffic on Google's high performance backbone for the longest possible distance, for reduced latency and improved throughput. Please refer to the document for more information.