Wondering if this is possible? I'd like to have 2 debezium connector processes (with different server ids) pointing to the same MySQL database and receiving the data as if they are workers sharing the load. Currently I observe the behaviour of pub/sub where both gets a copy of the same event. If I use the same server.id, then the first process dies complaining that the process with the same server id has been started.
I use Kafka as a store.
Wondering if this is possible or Debezium connectors are always meant to be "single process" and not horizontally scale?
Just a thought here. MySql DB delta bin log must be read sequentially by a single task.To distribute the tasks against multiple tables use multiple connectors reading from different tables from the same DB
https://debezium.io/documentation/reference/stable/connectors/mysql.html
tasks.max : 1: The maximum number of tasks that should be created for this connector. The MySQL connector always uses a single task and therefore does not use this value, so the default is always acceptable.
Ref : https://debezium.io/documentation/reference/stable/connectors/mysql.html
You could have HA using same group id , when one of the connect servers go down the other connector would pick up where the previous connector left off.