Load data from csv in airflow docker container to snowflake DB

15 Views Asked by At

I am running airflow in docker. I have an Extract task that extracts data from some API and stores it to a csv file in my docker container (path is /opt/airflow/raw/data.csv

I need to add a load task in my DAG to load the data from the csv file in docker container to a snowflake table. The code for my dag task is

from airflow import DAG
from airflow.providers.snowflake.operators.snowflake import SnowflakeOperator

load_task = SnowflakeOperator(
        task_id='load_to_snowflake',
        snowflake_conn_id='snowflake_conn',
        sql=f"""
        COPY INTO rates
        FROM 'file:///opt/airflow/raw/data.csv'
        FILE_FORMAT=(TYPE=CSV SKIP_HEADER=1)
        """,
        autocommit=True,
)

I get error that the url file:///opt/airflow/raw/data.csv is invalid. Please any ideas?

0

There are 0 best solutions below