Can't run Airflow standalone on M2

64 Views Asked by At

I've installed pyenv and poetry, and using these too i've created a poetry environment in which i could run airflow

[tool.poetry.dependencies]
python = "~3.11"
jupyterlab = "^4.0.7"
langchain = "^0.0.327"
llama-index = "^0.8.56"
pypdf = "^3.17.0"
python-dotenv = "^1.0.0"
botocore = "1.31.76"
boto3 = "1.28.76"
boto3-stubs = { version = "1.28.76", extras = ["essential"] }


[tool.poetry.group.dev.dependencies]
isort = "^5.12.0"
mypy = "^1.6.1"
flake8 = "^6.1.0"
black = "^23.10.1"

[tool.poetry.group.standalone]
optional = true

[tool.poetry.group.standalone.dependencies]
apache-airflow = "^2.7.2"
connexion = "^2.14.2"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Yet, after installing and ensuring i'm running airflow inside the shell, i'm running into a set of errors:

airflow standalone
standalone | Starting Airflow Standalone
standalone | Checking database is initialized
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
ERROR [airflow.models.dagbag.DagBag] Failed to import: /opt/homebrew/lib/python3.11/site-packages/airflow/example_dags/example_local_kubernetes_executor.py
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.11/site-packages/airflow/models/dagbag.py", line 342, in parse
    loader.exec_module(new_module)
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/opt/homebrew/lib/python3.11/site-packages/airflow/example_dags/example_local_kubernetes_executor.py", line 28, in <module>
    from airflow.decorators import task
  File "/opt/homebrew/lib/python3.11/site-packages/airflow/decorators/__init__.py", line 22, in <module>
    from airflow.decorators.branch_python import branch_task
  File "/opt/homebrew/lib/python3.11/site-packages/airflow/decorators/branch_python.py", line 22, in <module>
    from airflow.decorators.python import _PythonDecoratedOperator
  File "/opt/homebrew/lib/python3.11/site-packages/airflow/decorators/python.py", line 22, in <module>
    from airflow.operators.python import PythonOperator
  File "/opt/homebrew/lib/python3.11/site-packages/airflow/operators/python.py", line 51, in <module>
    from airflow.utils.process_utils import execute_in_subprocess
  File "/opt/homebrew/lib/python3.11/site-packages/airflow/utils/process_utils.py", line 40, in <module>
    import psutil
  File "/Users/B242PN/Library/Python/3.11/lib/python/site-packages/psutil/__init__.py", line 123, in <module>
    from . import _psosx as _psplatform
  File "/Users/B242PN/Library/Python/3.11/lib/python/site-packages/psutil/_psosx.py", line 13, in <module>
    from . import _psposix
  File "/Users/B242PN/Library/Python/3.11/lib/python/site-packages/psutil/_psposix.py", line 28, in <module>
    from . import _psutil_osx
ImportError: dlopen(/Users/B242PN/Library/Python/3.11/lib/python/site-packages/psutil/_psutil_osx.abi3.so, 0x0002): tried: '/Users/B242PN/Library/Python/3.11/lib/python/site-packages/psutil/_psutil_osx.abi3.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/B242PN/Library/Python/3.11/lib/python/site-packages/psutil/_psutil_osx.abi3.so' (no such file), '/Users/B242PN/Library/Python/3.11/lib/python/site-packages/psutil/_psutil_osx.abi3.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))

I'm not understanding where this comes from. I've initially installed poetry using the curl command provided in their documentation, then moved to the poetry on homebrew. It also produced the same errors as above, so when I came back to the initial installation method, I still ran into the same problems.

After doing some research, it seems that Airflow should be working on M2 when above 2.3, which is the case. I've also tried reinstalling psutil, but it didn't help. Very confused.

Am I missing something?

Thanks

0

There are 0 best solutions below