I've installed pyenv and poetry, and using these too i've created a poetry environment in which i could run airflow
[tool.poetry.dependencies]
python = "~3.11"
jupyterlab = "^4.0.7"
langchain = "^0.0.327"
llama-index = "^0.8.56"
pypdf = "^3.17.0"
python-dotenv = "^1.0.0"
botocore = "1.31.76"
boto3 = "1.28.76"
boto3-stubs = { version = "1.28.76", extras = ["essential"] }
[tool.poetry.group.dev.dependencies]
isort = "^5.12.0"
mypy = "^1.6.1"
flake8 = "^6.1.0"
black = "^23.10.1"
[tool.poetry.group.standalone]
optional = true
[tool.poetry.group.standalone.dependencies]
apache-airflow = "^2.7.2"
connexion = "^2.14.2"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
Yet, after installing and ensuring i'm running airflow inside the shell, i'm running into a set of errors:
airflow standalone
standalone | Starting Airflow Standalone
standalone | Checking database is initialized
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
ERROR [airflow.models.dagbag.DagBag] Failed to import: /opt/homebrew/lib/python3.11/site-packages/airflow/example_dags/example_local_kubernetes_executor.py
Traceback (most recent call last):
File "/opt/homebrew/lib/python3.11/site-packages/airflow/models/dagbag.py", line 342, in parse
loader.exec_module(new_module)
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/opt/homebrew/lib/python3.11/site-packages/airflow/example_dags/example_local_kubernetes_executor.py", line 28, in <module>
from airflow.decorators import task
File "/opt/homebrew/lib/python3.11/site-packages/airflow/decorators/__init__.py", line 22, in <module>
from airflow.decorators.branch_python import branch_task
File "/opt/homebrew/lib/python3.11/site-packages/airflow/decorators/branch_python.py", line 22, in <module>
from airflow.decorators.python import _PythonDecoratedOperator
File "/opt/homebrew/lib/python3.11/site-packages/airflow/decorators/python.py", line 22, in <module>
from airflow.operators.python import PythonOperator
File "/opt/homebrew/lib/python3.11/site-packages/airflow/operators/python.py", line 51, in <module>
from airflow.utils.process_utils import execute_in_subprocess
File "/opt/homebrew/lib/python3.11/site-packages/airflow/utils/process_utils.py", line 40, in <module>
import psutil
File "/Users/B242PN/Library/Python/3.11/lib/python/site-packages/psutil/__init__.py", line 123, in <module>
from . import _psosx as _psplatform
File "/Users/B242PN/Library/Python/3.11/lib/python/site-packages/psutil/_psosx.py", line 13, in <module>
from . import _psposix
File "/Users/B242PN/Library/Python/3.11/lib/python/site-packages/psutil/_psposix.py", line 28, in <module>
from . import _psutil_osx
ImportError: dlopen(/Users/B242PN/Library/Python/3.11/lib/python/site-packages/psutil/_psutil_osx.abi3.so, 0x0002): tried: '/Users/B242PN/Library/Python/3.11/lib/python/site-packages/psutil/_psutil_osx.abi3.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/B242PN/Library/Python/3.11/lib/python/site-packages/psutil/_psutil_osx.abi3.so' (no such file), '/Users/B242PN/Library/Python/3.11/lib/python/site-packages/psutil/_psutil_osx.abi3.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))
I'm not understanding where this comes from. I've initially installed poetry using the curl command provided in their documentation, then moved to the poetry on homebrew. It also produced the same errors as above, so when I came back to the initial installation method, I still ran into the same problems.
After doing some research, it seems that Airflow should be working on M2 when above 2.3, which is the case. I've also tried reinstalling psutil, but it didn't help. Very confused.
Am I missing something?
Thanks