I have two Kafka topics (topic_A and topic_B) with identical partitioning (4 partitions) and similar data structures. Both topics require the same processing work.
Currently, I'm considering a single Kafka Connect Sink connector to write data to BigQuery, with tasks.max set to 8. This setup will run within a Kubernetes (K8s) cluster configured to scale up pods/workers based on topic latency, pod CPU, or pod memory usage.
Are there advantages to using two separate Sink connectors (one per topic) instead of a single connector for this scenario? If so, what are they? I'm primarily interested in factors like performance, fault tolerance, scaling flexibility, and overall management.
Two connectors can be given different configurations, if needed. Partitions isn't the only thing that matters. Authentication, serialization format, record sizes, etc will need to be similar as well.
Fault tolerance is also a factor since the tasks can rebalance and any failure could halt the complete task, so stopping it for all topics being consumed