Export from AlloyDB in Composer Airflow - SSH connection difficulties

81 Views Asked by At

The issue I am encountering is how to create an SSH tunnel using IAP for AlloyDB export in Airflow hosted on Google Composer.

I am replacing a CloudSQL connection with AlloyDB. The composer environment is on a different network than my AlloyDB instance, so I know I need to use a bastion VM as listed here.

On local, I am able to port forward to my alloydb-proxy-vm and run my Airflow process. I am portforwarding in my Terminal. Thus, my VM should be properly configured.

To build the functionality into my Airflow, I tried the BashOperator like this:

    SSH_START = BashOperator(
        task_id='create_ssh_tunnel',
        bash_command="""
                gcloud auth activate-service-account --key-file=/opt/airflow/np-serviceaccount.json
                gcloud config set project np-myproject
                gcloud compute ssh alloydb-psql-client \
                       --tunnel-through-iap \
                       --zone=us-central1-b \
                       --ssh-flag="-T -L 0.0.0.0:5432:localhost:5432"
                """,
        dag=dag
    )


SSH_START >> ALLOYDB_EXPORT (using PostgresTOGCSOperator)

However, the issue with this is that I am simply opening and closing a tunnel as a task, I am not enabling the task that needs the connection to use it.

Should I be looking to use the ComputeEngineSSHHook or something similar instead?

Any advice would be greatly appreciated.

0

There are 0 best solutions below