PDI transformation does not send messages to Kafka server

510 Views Asked by At

I have a transformation in Pentaho Data Integration (PDI) that makes a query to NetSuite, builds JSON strings for each row and finally these strings are sent to Kafka. This is the transformation:

PDI Transformation

When I test the transform against my local Kafka it works like a charm, as you can see below:

Messages received in local Kafka

The problem is when I substitute the connection parameters for those of an AWS EC2 instance where I have Kafka as well. The problem is that the transformation does not give errors, but the messages do not reach Kafka, as can be seen here:

No errors in transformation No messages received in Kafka server

This is the configuration of the Kafka Producer step of the transformation:

Kafka Producer step config

The strange thing is that although it does not send the messages to Kafka, it seems that it does connect to the server because the combobox is displayed with the names of the topics that I have:

Kafka server topics displayed

In addition, this error is observed in the PDI terminal:

ERROR [NamedClusterServiceLocatorImpl] Could not find service for interface org.pentaho.hadoop.shim.api.jaas.JaasConfigService associated with named cluster null

PDI terminal showing error

Which doesn't make sense to me because I'm using a direct connection and not a connection to a Hadoop Cluster.

So I wanted to ask the members of this community if anyone has used POIs to send messages to Kafka and if they had to make configurations in POI or Slack to achieve it, since I cannot think what could be happening.

Thanks in advance for any ideas or comments to help me solve this!

0

There are 0 best solutions below