Hey there Overflowers!
I'm working on a small app that triggers a Cloud Function every time a message is published to a PubSub topic. The code inside the function is working (apparentely) fine, but I'm experimenting some issues with the publishing process.
For some reason, when posting messages a warning log from the Cloud Function returns: "POST 429 [...] The request was aborted because there was no available instance".
I wonder to know what it means and how to deal with it, if anyone could help. I'm not sure how this can be solved from my code.
Thanks everyone!
Edit: The function memory is limited to 256MB, with a 60 secs timeout.
Edit 2: I've tested removing instances limitations but the service keeps failing (returning a POST 500 or a POST 429 errors).
I have another question, my Cloud Functions are returning a timeout sometimes. I'm pretty sure the origin of the problem is in PubSub, the one metioned before, and these CF corresponds to the failing POST requests. In the majority of the cases the functions are executed properly and inside the time range.
My questions: Why these functions are being executed if the message wasn't posted properly at first (maybe it is retried?) and, more important, why these functions are reaching their timeout limit (they shouldn't trigger until a message is published properly, right)?
Edit 3: I've been reading the Google Cloud Functions documentation and I've found my problem already documented here: https://cloud.google.com/functions/docs/troubleshooting#scalability
As Guillaume said (thanks again!) it seems to be a scaling issue. The functions are being triggered so fast that they have no enough time to scale instances. I'm gonna try handling the flow of the incoming messages to be slower while ramping.
The notes from this section could help too: https://cloud.google.com/functions/docs/concepts/exec#auto-scaling_and_concurrency
Note: An extremely rapid increase in inbound traffic can intermittently cause some requests to fail with an HTTP code of 500. This is because the requests are timing out in the pending queue while waiting for new instances to be created. Configuring your workload so that it ramps traffic up gradually over the course of a minute can help with this issue. Also see Troubleshooting Cloud Functions for more information.
Thanks again everyone!
You have throttled the Cloud Functions creation to 4 instances. And Cloud Functions can only process 1 request (1 message in your case) at a time.
Therefore, if you have more than 4 messages in the same time in PubSub, you will have this 429 HTTP errors.
However, it's not a problem if your use case is not time sensitive. Activate the retry in your Cloud Functions deployment, and the message will be retried later on.
If you want that the message is processed ASAP, consider to increase the Cloud Functions max instance limit, or use product that accept parallel processing (Cloud Run and App Engine for instance)