I'm new to both docker and airflow and I'm trying to create an airflow pipeline in docker on my windows desktop. I have used the latest airflow image and extended it to include some additional python libraries that are needed. All the libraries were installed, image extended and container up and running.
Next, I write the .py file which contains all DAG's and have imported the following modules.
from airflow import DAG
from airflow.operators.bash import BashOperator
import time as tm
import pickle, os, datetime, json, requests, locale
from datetime import datetime, date, time, timedelta
from yahoo_fin import options as op
import yfinance as yf
import pandas as pd, numpy as np
from polygon import RESTClient
from polygon.rest.models import Sort
from urllib3 import HTTPResponse
After saving this file, I see the following error message on airflow UI:
Broken DAG: [/opt/airflow/dags/our_first_dag.py] Traceback (most recent call last): File "", line 219, in _call_with_frames_removed File "/opt/airflow/dags/our_first_dag.py", line 9, in from polygon import RESTClient ImportError: cannot import name 'RESTClient' from 'polygon' (/home/airflow/.local/lib/python3.8/site-packages/polygon/init.py)
From this message it appears that python is unable to import RESTClient from Polygon module, so I check if this package is installed in the image by by going to the container terminal (from docker UI) confirm that all the packages I added while extending the image are actually installed. I also tried running the line which produced error i.e. "from polygon import RESTClient" in the local python and it worked perfectly fine.
I'd appreciate any suggestions on how to fix this.
Below are the contents of my directory:
dockerfile
FROM apache/airflow:2.7.1
COPY requirements.txt /
USER airflow
RUN pip install -r /requirements.txt
USER root
requirements.txt
polygon
html5lib
yahoo_fin
yfinance
polygon-api-client
.env
AIRFLOW_UID=50000
docker-compose.yaml https://pastebin.com/SPNsk1f4