how can i put all tabels changes in one streams in debezium&redis-stream

47 Views Asked by At

I want to monitor one postgres database's changes with debezium-server and put all these changes into redis stream.But after my test i can only xread one table's change in one command, like xread block 1000000 streams 'topix_prefix_name'.'schema_name'.'table_name'. my project uses python. thanks a lot. Besides,there is another question that why i can use topix.prefix even i didn't use kafka?

It's useless to use command xread block 1000000 streams 'topix_prefix_name'.'schema_name',even I have configure the 'application.properties' file as follow:

debezium.sink.type=redis
debezium.sink.redis.address=redis:6379
debezium.source.connector.class=io.debezium.connector.postgresql.PostgresConnector
debezium.source.offset.storage.file.filename=data/offsets.dat
debezium.source.offset.flush.interval.ms=0
debezium.source.database.hostname=postgres
debezium.source.database.port=5432
debezium.source.database.user=postgres
debezium.source.database.password=postgres
debezium.source.database.dbname=mydb
debezium.source.topic.prefix=mydb
debezium.source.schema.include.list=public
debezium.source.plugin.name=pgoutput
debezium.source.decimal.handling.mode=double
1

There are 1 best solutions below

0
zalmane On

In order to route all of your records to one topic, take a look at topic routing: https://debezium.io/documentation/reference/stable/transformations/topic-routing.html

For example, to route all data to the same topic, mytopic:

debezium.transforms=Reroute
debezium.transforms.Reroute.type=io.debezium.transforms.ByLogicalTableRouter
debezium.transforms.Reroute.topic.regex=(.*)
debezium.transforms.Reroute.topic.replacement=$1customers_all_shards

You can also use this to route selectively. This example routes all shards of a table to the same topic:

debezium.transforms=Reroute
debezium.transforms.Reroute.type=io.debezium.transforms.ByLogicalTableRouter
debezium.transforms.Reroute.topic.regex=(.*)customers_shard(.*)
debezium.transforms.Reroute.topic.replacement=$1customers_all_shards