Module not found Faker, db_connect causing DAG to break in apache airflow

Question

Broken DAG: [/opt/airflow/dags/etl_pipeline_dag.py] Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/airflow/dags/etl_pipeline_dag.py", line 4, in <module>
    from db_connect import PG
ModuleNotFoundError: No module named 'db_connect'

here db_connect.py is my script which is located in the same directory as dag folder
and etl_pipeline_dag has dag operations where i have imported Faker(python package) to generate fake data. getting the same error even after installing using pip in the same environment where i installed airflow using pip too

I used Docker to run the airflow container and kept the Docker Compose file as default from the official documentation site.

I tried making Dockerfile and building the image for the same, but it also didn't work.

Can anyone help me with this?

Answer 1

I found the solution to it, it lies in creation of Dockerfile with commands like:

FROM apache/airflow:2.7.1 RUN pip install faker COPY db_connect.py /opt/airflow/ COPY config.py /opt/airflow/

and the saving it and heading to the terminal to type: docker build . --tag {name_of_your_image} hit enter to run it, after this go to docker-compose.yaml file and change : image: ${AIRFLOW_IMAGE_NAME:-{name_of_your_image}}

run docker-compose up -d and you won't face this issue anymore

Note: you can add more python libraries and custom modules to Dockerfile but be sure to run command: build . --tag {name_of_your_image} and docker compose up -d