Meltano ETL not deleting rows in destination PostgreSQL database after deleting from source PostgreSQL database

Question

I am using Meltano for ETL to extract data from a source PostgreSQL database using tap-postgres (MeltanoLabs variant) and load it into a destination PostgreSQL database using target-postgres (TransferWise variant). However, I have noticed that when I delete rows from the source PostgreSQL database, the deleted rows are not removed from the destination PostgreSQL database.

I have tried using the --full-refresh option with the meltano run command, but this does not seem to delete the rows from the destination database. Is there a way to reset the data in the destination database before running the job with Meltano?

Here is the command I am using to perform the ETL:
meltano run tap-postgresql target-postgres --full-refresh

I have verified that the rows have been deleted from the source PostgreSQL database by running a SQL query, but the deleted rows still exist in the destination PostgreSQL database after running the job.

Any help or suggestions would be greatly appreciated. Thank you.

Answer 1

You'll need to turn on the hard_delete option for the tap and the add_metadata_columns See https://github.com/transferwise/pipelinewise-target-postgres