I have set up a PubSub BigQuery Subscription. My topic has a schema, and I am able to succesfully write to the BigQuery table.
I have been stress testing the workflow to ensure that all messages sent to the topic are succesfully written to BigQuery. I have been doing batches of 500k messages via Python/PubSub client library. My results have been inconsistent. Sometimes all 500k records make it into BigQuery, other times they do not. Typically > 98% of records get written. I also have a side subscription hooked to the topic that does not consume messages, I use it to simply verify that all messages I sent to the topic made it. This side subscription ALWAYS recieves every single message I send to it - every single time.
Additionally, I have a deadletter topic that has the SAME exact schema as the topic that is writing to BigQuery - not once have the "missing" records from BigQuery showed up in the deadletter topic.
I have tried adjusting all of the settings possible in the BigQuery subscription - but still seeing inconsistent results.
I would expect that every single record that comes through the topic would either be written to BigQuery or end up in the dead letter sub.
Has anyone had this kind of issue before? What am I missing?
Thank you in advance.
For such an issue, you would need to enter a support ticket as it will require looking into the backend details of your subscription and table. In general, Pub/Sub guarantees that all well-formed messages that match the table schema will be written to the table when using a BigQuery subscription.