table.include.list configuration parameter not working in Debezium Postgres Connector

Question

I am using Debezium Postgres connector to capture changes in postgres tables.

The documentation for this connector https://debezium.io/documentation/reference/connectors/postgresql.html

mentions a configuration parameter

table.include.list

However when I set the value of this parameter to 'config.abc'. Even after that changes from both tables in config schema (namely abc and def) are getting streamed.

The reason I want to do this is that I want to create 2 separate connectors for each of the 2 tables to split the load and faster change data streaming.

Is this a known issue ? Anyway to overcome this ?

Answer 1

The same problem here (Debezium Version: 1.1.0.Final and postgresql:11.13.0). Below is my configuration:

{
  "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
  "transforms.unwrap.delete.handling.mode": "rewrite",
  "slot.name": "debezium_planning",
  "transforms": "unwrap,extractInt",
  "include.schema.changes": "false",
  "decimal.handling.mode": "string",
  "database.schema": "partsmaster",
  "transforms.unwrap.drop.tombstones": "false",
  "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
  "value.converter": "org.apache.kafka.connect.json.JsonConverter",
  "key.converter": "org.apache.kafka.connect.converters.LongConverter",
  "database.user": "************",
  "database.dbname": "smartng-db",
  "transforms.extractInt.type": "org.apache.kafka.connect.transforms.ExtractField$Key",
  "database.server.name": "ASLog.m3d.smartng-db",
  "database.port": "***********",
  "plugin.name": "pgoutput",
  "column.exclude.list": "create_date, create_user, change_date, change_user",
  "transforms.extractInt.field": "planning_id",
  "database.hostname": "************",
  "database.password": "*************",
  "name": "ASLog.m3d.source-postgres-smartng-planning",
  "transforms.unwrap.add.fields": "op,table,lsn,source.ts_ms",
  "table.include.list": "partsmaster.planning",
  "snapshot.mode": "never"
}

A change in another table partsmaster.basic is causing this connector to fail because the attribute planning_id is not available in table partsmaster.basic.

I need separate connectors for each table. table.include.list is not working. Neither table.exclude.list does.

My last resort would be to use a separate Postgres Publication for each connector. But I still hope I can find an immediate configuration solution here or be pointed out at what I am missing. In a previous version I have used table.whitelist without problems.

Solution:

1. create seperate publications to each table:

   CREATE PUBLICATION dbz_planning FOR TABLE partsmaster.planning;

2. drop previous replication slot:

   select pg_drop_replication_slot('debezium_planning');

3. Change your connector configurations:

   { "slot.name": "dbz_planning", "publication.name": "planning_publication" }

et voilà!

Answer 2

Debezium documentation says

Thus, you’ll need a unique publication for each connector, unless the connectors work on same set of tables