Include new tables with data to existing Debezium connector

Question

Objective:

Need to add a new table to existing Debezium MySQL connector version 1.1.1.Final. The data from the table should be populated to the topic after the connector restart with updated configs(new tables in the table.whitelist).

Problem:

There is no clear strategy to add a new table(s) which will be added to the Kafka with its data after the initial snapshot was taken (the previous version of the connector is working). We are looking for something similar to snapshot.select.statement.overrides only for the working connector. snapshot.select.statement.overrides doesn't work when the snapshot is already created and the connector is restarted with new tables.

Possible solution:

Create and run new connector for tables which should be initialized with data(snapshot.mode=initial). Disadvantage: every release may include a new connector.

OR

1. Create migration connector(v2) for the tables (with configs snapshot.mode=initial and new database.history.kafka.topic)
2. Delete connector v2 after snapshotting completed
3. Remove database.history.kafka.topic for the previous connector(v1)
4. Update connector(v1) one with tables from the connector(v2) - put them to table.whitelist
5. Change snapshot.mode to schema_only_recovery and start the connector(v1)

Question:

Which way is official? Maybe am I missing something and there is more easiest way than the second?

UPD: Per @Matar comment and some investigation, it looks like no need to remove database.history.kafka.topic because, in the default configuration, it collects all DDL from the DB anyway.

Answer 1

With the latest version of Debezium Server, you can add the following config value

debezium.snapshot.new.tables=parallel

In case If you are using Debezium, you can try this config value

snapshot.new.tables=parallel

Note: Debeziyum Server is the one that supports Kinesis, Google Pub sub, and Apache Pulsar. I am using that and its configuration is a bit different. I had to prepend "debezium" before each item

Once this configuration is added, any addition to tables.whitelist, For these additional tables Debezium will create snapshots.