Install custom package (wheel) in MWAA

Question

We are trying to install python package that can be shipped in plugins.zip for MWAA environment and added the reference in requirements.txt as described in the docs but we are still seeing the error file not found.

directory structure

├── README.md
├── dags
│   └── dummy.py
├── plugins
│   └── dummy-wheel-0.1.0-py3-none-any.whl
└── requirements.txt

The contents of requirements.txt /usr/local/airflow/plugins/dummy-wheel-0.1.0-py3-none-any.whl

From here I have just created a zip file containing dummy-wheel-0.1.0-py3-none-any.whl and push it to S3://<bucket_name>/plugins.zip

Followed instructions here https://aws.amazon.com/premiumsupport/knowledge-center/mwaa-install-custom-packages/

What am I missing?

Answer 1

You may utilize BashOperator to list the directory /usr/local/airflow/plugins/ to verify whether the file is present there. While zipping, the whl file is supposed to be in the root path for it to be available in /usr/local/airflow/plugins/. Whereas if zipping a folder (eg. plugins), the extracted path will vary accordingly (eg. /usr/local/airflow/plugins/plugins/).

Answer 2

@mad_

Yeah I would try listing the contents of /usr/local/airflow/plugins first:

ls_airflow_plugins = BashOperator(
    task_id="ls_airflow_plugins",
    bash_command="ls -laR /usr/local/airflow/plugins",
    dag=dag,
    priority_weight=300,
)

Also, some things to try are:

put the actual plugins.zip file in a plugins/ folder that is at the same level as dag/ folder.

Before zipping the whl file, make sure you have given 755 file permissions (chmod 755 command)

at the top of your requirements.txt file, include:

--find-links /usr/local/airflow/plugins
--no-index

Hope one of these fixes it, lmk of questions

Answer 3

@mad_ Can you explain where you are seeing the file not found error?

If you zip packages and then edit/save your environment to point to the plugins.zip file, enable logging in the monitoring section before you re-deploy. The scheduler CloudWatch log will list packages that were installed (or errors that occurred). That may point you in the right direction.

The other thing to check is dependencies. If your package depends on a certain version of a package that MWAA already installs, you may need to include that dependency too.