I'm using java-pubsub-group-kafka-connector to send messages from AWS MSK to GCP pub/sub. I created AWs MSK connector and status is running. But when send message to AWS MSK, after some time see this error in connector logs:
Also found this error:
Offset commit failed, rewinding to last committed offsets
Maybe, there is another solution to send messages from MSK to pub/sub?
Amazon MSK stands for Amazon Managed Streaming for Apache Kafka.
In other words, MSK is Kafka, when you interact with it using Apache Kafka APIs - consumer, producer or admin.
The provided link leads to a sink and source connectors that can connect MSK (or any other Kafka) cluster and GCP Pub-Sub. Either sink or source connector should be deployed on Kafka Connect cluster, which needs to be in close proximity with hosting Kafka cluster. This can be any kind of self-managed Kafka Connect on EC2, or any container solution, as long as you know how to deal with. Alternatively, these connectors can run on MSK Connect, which is a managed Kafka Connect.
What you will need:
1.Ensure connectivity between VPC/subnets where Kafka Connect with connectors runs, to the GCP Endpoint
2.Start Kafka Connect cluster which is using MSK as a hosting Kafka cluster. Copy binaries of those connectors to a plugin dir defined in worker configuration. Under MSK Connect - create a custom plugin
3.Create connectors by using rest endpoint of self-managed kafka connect or by using MSK Connect API
4.Pray: Observe, monitor and troubleshoot until it's successfully pulling/pushing some data