I want to cache Poetry virtual environments between builds of my Python application. I have configured the pipeline so that the virtual environment is recreated every time pyproject.toml changes (poetry.lock is not comitted):
default:
cache:
- key:
files:
- lego/pyproject.toml
paths:
- lego/.venv
- artifacts/build/layer/build/.venv
- paths:
- artifacts/pypoetry
The virtual environment is created as follows:
poetry config virtualenvs.in-project true
poetry install
This works locally, but it fails in GitLab CI/CD on the second pipeline run:
$ LOG_LEVEL=ERROR poetry run pytest tests -n auto -v --cov=./src --junitxml=lego-report.xml
ImportError while loading conftest '/builds/5565557/lddpro-bff/lego/tests/conftest.py'.
tests/conftest.py:22: in <module>
from lego.common.aws import S3, EventBridge, SSM, Secrets
E ModuleNotFoundError: No module named 'lego'
The reason is that pytest cannot locate the root package (lego). However, I have verified that the root package is located in the virtual environment (.venv):
$ cat .venv/lib/python*/site-packages/lego.pth
/builds/5565557/lddpro-bff/lego/src
Here lego.pth includes the current job ID in the path.
How can I properly cache Poetry virtual environments between CI/CD pipeline runs?
Job configuration:
lego:
extends: .unittest
variables:
REPORT_NAME: lego-report.xml
script:
- cd lego
- poetry config virtualenvs.in-project true && poetry install
- poetry env info
- env LOG_LEVEL=ERROR poetry run pytest tests -n auto -v --cov=./src --junitxml=${REPORT_NAME}
- mv -f ${REPORT_NAME} ${CI_PROJECT_DIR}/ || echo ""
- cd ${CI_PROJECT_DIR}
- mkdir -p artifacts
- cd tools
- . ./build_deploy.sh --command=build --debug
artifacts:
paths:
- artifacts/lego-function.zip
- artifacts/lego-layer.zip
reports:
junit: ${REPORT_NAME}
expire_in: 7 days
rules:
- if: $CI_COMMIT_BRANCH && $CI_PIPELINE_SOURCE != "merge_request_event"
- if: $CI_PIPELINE_SOURCE == "merge_request_event"