I am experiencing an issue with Google Cloud Pub/Sub where approximately 20% of my messages are being redelivered, even though they have been successfully processed and acknowledged by a Cloud Run service. I have verified that the acknowledgments are successful within a significantly shorter timeframe than my configured acknowledgment deadline.
Messages are pushed from a Pub/Sub topic to a Cloud Run service via a push subscription.
My acknowledgment deadline is set to 300 seconds. Out of 174 messages pushed, about 30 were redelivered despite Cloud Run responding with HTTP 200 status codes for all messages, within 30 seconds (not fast but not above 300s).
I initially suspected that Pub/Sub did not receive some HTTP 200 responses, causing unintended redeliveries but the monitoring indicates no messages exceeded the acknowledgment deadline, as the "Expired ack deadlines count" metric showed zero for the period.
I understand that Pub/Sub guarantees at-least-once delivery, but the observed redelivery rate seems unusually high.
All affected messages were confirmed to be successfully processed by Cloud Run, with logs indicating receipt and processing completion (response code 200) well before the acknowledgment deadline.
Any insight is welcome.