Datastream - the costs of root partitioned table from PostgreSQL to BigQuery

94 Views Asked by At

I have a very large partitioned table in PostgreSQL. It is replicated to BigQuery using Google Datastream as separate partitioned tables (as described in docs). Right now I have a separate table in BigQuery for each partition of the PostgreSQL table.

my_table_name_2022_01
my_table_name_2022_02
my_table_name_2022_03
...

I want to switch the configuration to a single root table. I am concerned that it may increase the costs of stream table updates in BigQuery.

If I configure a stream with a single root table, will Datastream occasionally scan all BigQuery table partitions during streaming? I am interested in the case when there will be only INSERT and UPDATE operations for the Postgres table. But the case including DELETE, TRUNCATE operations is also interesting.

0

There are 0 best solutions below