Kafka mongo db source connector not working

Question

Hi in my POC I am using both the sink and the source mongodb connector.
The sink connector works fine. But the source connector does not push data into the resultant topic. The objective is to push full documents of all changes (Insert and Update) in a collection call 'request'.

Below is the code.

curl -X PUT http://localhost:8083/connectors/source-mongodb-request/config -H "Content-Type: application/json" -d '{
      "tasks.max":1,
      "connector.class":"com.mongodb.kafka.connect.MongoSourceConnector",
      "key.converter":"org.apache.kafka.connect.storage.StringConverter",
      "value.converter":"org.apache.kafka.connect.storage.StringConverter",
      "connection.uri":"mongodb://localhost:27017",
      "pipeline":"[]",
      "database":"proj",
      "publish.full.document.only":"true",
      "collection":"request",
      "topic.prefix": ""
}'

No messages are getting pushed to proj.request topic. The topic gets created once I insert a record in the collection 'request'.

Would be great to get help on this, as its a make or break task for the POC. Things work fine n the connectors on confluent cloud. But its the on premise set up on which I need to get this working.

Answer 1

make sure you have a valid pipeline - stages included in your properties file such as this one

"pipeline":" [{"$match":{"type":{"$in"["insert","update","replace"]}}}]",

Refer : https://docs.mongodb.com/manual/reference/operator/aggregation-pipeline/