Is it possible to set Kafka to receive a message in the topic with the name of the file that is uploaded to Google Storage?

Question

The connector allows me to read only json, avro, bytes and string formats. But if a pdf file arrives that then needs to be processed by a script that is executed via an image associated with a cloud run, do you have any ideas on how to do it?

(without using pub sub to communicate with kafka)

Answer 1

I would recommend the following solution:

1. Use kafka connect to list gcs files specifically the kafka-connect-file-pulse connectors. https://streamthoughts.github.io/kafka-connect-file-pulse/docs/getting-started/

2. You can play with the metadata from an uploaded file that will be published to Kafka and then use that filename message from a Kafka consumer from a client subscriber running on Cloudrun.

hope that help you