Airbyte pg to pg failing exactly after 8 gb of data transfer

48 Views Asked by At

I am using latest Airbyte source and destination postgres connectors at the time of writing this.

When 8 gb of data is transferred, soon after that the sync fails with no error in logs.

I tried this 3-4 times, same behaviour each time.

I set JOB_MAIN_CONTAINER_MEMORY* enva to 3 gbs each and set mem_limit on airbyte-worker to 5 gb as without this it was eating all ram and going till 8 gb and finally freezing the ec2.

Please help to fix this issue

Thanks in advance!!!

0

There are 0 best solutions below