Task becomes UNASSIGNED for Debezium MySQL source connector

327 Views Asked by At

I am using debezium 1.9. I created a connector using below config

{ 
    "name": "user_management_db-connector-5", 
    "config": { 
        "connector.class": "io.debezium.connector.mysql.MySqlConnector", 
        "tasks.max": "1", 
        "database.hostname": "XXXX", 
        "database.port": "3306", 
        "database.user": "XXX", 
        "database.password": "XXX", 
        "database.server.id": "12345", 
        "database.server.name": "ula-stg-db", 
        "database.include.list": "user_management_db", 
        "database.history.kafka.bootstrap.servers": "kafka.ulastg.xyz:9094,kafka.ulastg.xyz:9092", 
        "database.history.kafka.topic": "dbhistory.user_management_db",
        "snapshot.mode" : "schema_only", 
        "snapshot.locking.mode" : "none",
        "table.include.list": "user_management_db.user,user_management_db.store,user_management_db.store_type,user_management_db.user_segment,user_management_db.user_segment_mapping",
        "transforms":"Reroute", 
        "transforms.Reroute.type":"io.debezium.transforms.ByLogicalTableRouter", 
        "transforms.Reroute.topic.regex":"(.*)user_management_db(.+)", 
        "transforms.Reroute.topic.replacement":"$1cdc", 
        "key.converter": "org.apache.kafka.connect.json.JsonConverter", 
        "key.converter.schemas.enable": "false", 
        "value.converter": "org.apache.kafka.connect.json.JsonConverter", 
        "value.converter.schemas.enable": "false",
        "topic.creation.default.include": "ula-stg-db.+",  
        "topic.creation.default.partitions": 20,
        "topic.creation.default.replication.factor": 2, 
        "topic.creation.default.cleanup.policy": "delete",
        "topic.creation.default.delete.retention.ms": 300000,
        "errors.log.enable": true,
        "errors.log.include.messages" :true
    } 
}

The connector gets created and I can see events in the topic ula-stge-db.cdc

The problem is after some time ( approximately a day ) I see events stop getting populated. I do not see any error in connector logs.

It only throws a generic info in regular interval

2022-07-12 09:24:25,654 INFO || WorkerSourceTask{id=promo_management_db-connector-5-0} Either no records were produced by the task since the last offset commit, or every record has been filtered out by a transformation or dropped due to transformation or conversion errors. [org.apache.kafka.connect.runtime.WorkerSourceTask]

The connector status is now shown as below

{
  "name": "user_management_db-connector-5",
  "connector": {
    "state": "RUNNING",
    "worker_id": "172.31.65.156:8083"
  },
  "tasks": [
    {
      "id": 0,
      "state": "UNASSIGNED",
      "worker_id": "172.31.71.28:8083"
    }
  ],
  "type": "source"
}

How to debug further ?

P.S: I am connecting to AWS RDS MySql. And Kafka is hosted in an EC2.

0

There are 0 best solutions below