I have setup a MSK cluster using public subnets. I also have turned on public access for this MSK cluster. The cluster is up and running. I am trying to setup MongoDB Kafka Source Connector using MSK connect. I have selected IAM authentication and created IAM Role and corresponding policy for authentication. Below is my mongodb kafka source connector configuration
connector.class=com.mongodb.kafka.connect.MongoSourceConnector
tasks.max=1
connection.uri=mongodb+srv://XXXXX:[email protected]/user?authSource=admin&replicaSet=atlas-bftscy-shard-0&readPreference=primary
database=user
topic.prefix=mongo
poll.max.batch.size=1000
poll.await.time.ms=5000
pipeline=[]
batch.size=0
publish.full.document.only=true
mongo.errors.tolerance=all
mongo.errors.deadletterqueue.topic.name=data-ingestion-dlq-beta
offset.partition.name=mongodb-source-connector-beta.1
startup.mode=copy_existing
output.format.value=json
output.format.key=json
key.converter.schemas.enable=false
value.converter.schemas.enable=false
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=org.apache.kafka.connect.storage.StringConverter
change.stream.full.document=updateLookup
But when I am creating the connector it's getting failed due to the following error
[Worker-0aef29e0af9181e21] [2024-01-31 20:47:37,864] INFO Exception in monitor thread while connecting to server XXXXXXX.qricz.mongodb.net:27017 (org.mongodb.driver.cluster:76)
[Worker-0aef29e0af9181e21] com.mongodb.MongoSocketOpenException: Exception opening socket
[Worker-0aef29e0af9181e21] at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:70)
[Worker-0aef29e0af9181e21] at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:180)
[Worker-0aef29e0af9181e21] at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.lookupServerDescription(DefaultServerMonitor.java:193)
[Worker-0aef29e0af9181e21] at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:157)
[Worker-0aef29e0af9181e21] at java.base/java.lang.Thread.run(Thread.java:829)
[Worker-0aef29e0af9181e21] Caused by: java.net.SocketTimeoutException: connect timed out
[Worker-0aef29e0af9181e21] at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
[Worker-0aef29e0af9181e21] at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)
[Worker-0aef29e0af9181e21] at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)
[Worker-0aef29e0af9181e21] at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)
[Worker-0aef29e0af9181e21] at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[Worker-0aef29e0af9181e21] at java.base/java.net.Socket.connect(Socket.java:609)
[Worker-0aef29e0af9181e21] at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:305)
[Worker-0aef29e0af9181e21] at com.mongodb.internal.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:107)
[Worker-0aef29e0af9181e21] at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:79)
[Worker-0aef29e0af9181e21] at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:65)
[Worker-0aef29e0af9181e21] ... 4 more
What am I missing? Please help
Timeout exception is usually coming only from a network level issues.
Note, you don't have to have MSK in public subnets with public access enabled to access resources outside your VPC.
Possible issues: