Dagster executable missing from container

Question

I've been trying to create a container to run in Dagster cloud on an ECS-hybrid deployment model. I'm able to actually push the container to Dagster but I continually get this error:

dagster_cloud.workspace.ecs.client.EcsServiceError: ECS service failed because task arn:aws:ecs:ap-northeast-1:*****:task/Dagster-Cloud-my-cluster-Cluster/1d151e6d40b44588a4ed4446a949d44a failed: CannotStartContainerError: ResourceInitializationError: failed to create new container runtime task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "dagster": executable file not found in $PATH: unknown

Task logs:
runc create failed: unable to start container process: exec: "dagster": executable file not found in $PATH

For more information about the failure, check the ECS console for logs for task arn:aws:ecs:ap-northeast-1:*****:task/Dagster-Cloud-my-cluster-Cluster/1d151e6d40b44588a4ed4446a949d44a in cluster Dagster-Cloud-my-cluster-Cluster.
  File "/dagster-cloud/dagster_cloud/workspace/user_code_launcher/user_code_launcher.py", line 1304, in _reconcile
    self._wait_for_new_server_ready(
  File "/dagster-cloud/dagster_cloud/workspace/ecs/launcher.py", line 458, in _wait_for_new_server_ready
    task_arn = self.client.wait_for_new_service(
  File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 491, in wait_for_new_service
    return self.check_service_has_running_task(
  File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 607, in check_service_has_running_task
    self._raise_failed_task(task, container_name, logger)
  File "/dagster-cloud/dagster_cloud/workspace/ecs/client.py", line 526, in _raise_failed_task
    raise EcsServiceError(

I'm not sure why this is happening as I'm sure I'm installing dagster and dagster-cloud, as per the documentation. My Docker container looks like this:

###############################################################################
# Base container
###############################################################################
FROM python:3.11 AS base

# Define the environment variables necessary to work with poetry
ENV POETRY_VERSION=1.5.1 \
  POETRY_HOME="/opt/poetry" \
  POETRY_VIRTUALENVS_IN_PROJECT=true \
  POETRY_NO_INTERACTION=1

# Add the poetry bin to our path
ENV PATH="$POETRY_HOME/bin:$PATH"

###############################################################################
# Poetry installer container
###############################################################################
FROM base AS installer

# Install poetry
RUN curl -sSL https://install.python-poetry.org | python3 -

###############################################################################
# Container that actually builds the application
###############################################################################
FROM base AS builder

# Copy the poetry files from the poetry installer to this container
COPY --from=installer $POETRY_HOME $POETRY_HOME

# Describe the environment variables necessary to install our dependencies
ENV PYTHONFAULTHANDLER=1 \
  PYTHONUNBUFFERED=1 \
  PYTHONHASHSEED=random \
  PIP_NO_CACHE_DIR=off \
  PIP_DISABLE_PIP_VERSION_CHECK=on \
  PIP_DEFAULT_TIMEOUT=100

# Set the working directory to one we can install to easily
WORKDIR /app

# Copy the poetry.lock and pyproject.toml files first to ensure that dependencies
# are only installed when they're updated
COPY poetry.lock pyproject.toml /app/

# Copy in our SSH key so we can retrieve the shared GitHub repo
RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts

# Install our project dependencies
RUN --mount=type=ssh poetry install --only dagster --no-ansi --no-root

# Build the project
COPY . /app/
RUN poetry build

###############################################################################
# Create a runtime environment that's much smaller
###############################################################################
FROM python:3.11-alpine AS runtime

# Set the working directory to where Dagster will look for the application
WORKDIR /opt/dagster/app

# Copy the project wheel file
COPY --from=builder /app/dist/*.whl /

# Install the wheel file using pip and then install dagster and dagster-cloud
RUN pip install --no-cache-dir /*.whl \
  && rm -rf /*.whl

For this package, my pyproject.toml file lists my dependencies as follows:

[tool.poetry.group.dagster.dependencies]
dagster = "^1.5.9"
dagster-aws = "^0.21.9"
pendulum = "^2.1.2"
pandas = "^2.1.3"
openpyxl = "^3.1.2"
dagster-cloud = "^1.5.12"

So, both of these should be installed, but clearly they haven't been. What am I doing wrong here?

Answer 1

This container had two issues:

1. I was pushing across the virtual environment but that did not include the executables installed with some of the packages, which was causing the error.
2. Installing a wheel does not install the package's requirements.

To fix these issues, I used Poetry to construct a requirements.txt file and then used pip to install the requirements from it. As these included both dagster and dagster-cloud, the package ended up with both necessary executables.

###############################################################################
# Base container
###############################################################################
FROM python:3.11 AS base

# Define the environment variables necessary to work with poetry
ENV POETRY_VERSION=1.5.1 \
  POETRY_HOME="/opt/poetry" \
  POETRY_VIRTUALENVS_IN_PROJECT=true \
  POETRY_NO_INTERACTION=1

# Add the poetry bin to our path
ENV PATH="$POETRY_HOME/bin:$PATH"

###############################################################################
# Poetry installer container
###############################################################################
FROM base AS installer

# Install poetry
RUN curl -sSL https://install.python-poetry.org | python3 -

###############################################################################
# Container that actually builds the application
###############################################################################
FROM base AS builder

# Copy the poetry files from the poetry installer to this container
COPY --from=installer $POETRY_HOME $POETRY_HOME

# Describe the environment variables necessary to install our dependencies
ENV PYTHONFAULTHANDLER=1 \
  PYTHONUNBUFFERED=1 \
  PYTHONHASHSEED=random \
  PIP_NO_CACHE_DIR=off \
  PIP_DISABLE_PIP_VERSION_CHECK=on \
  PIP_DEFAULT_TIMEOUT=100

# Set the working directory to one we can install to easily
WORKDIR /app

# Copy the poetry.lock and pyproject.toml files first to ensure that dependencies
# are only installed when they're updated
COPY poetry.lock pyproject.toml /app/

# Create a requirements.txt file we can use
RUN poetry export --with dagster --without-hashes --with-credentials -f requirements.txt --output requirements.txt

###############################################################################
# Create a runtime environment that's much smaller
###############################################################################
FROM python:3.11 AS runtime

# Copy in our SSH key so we can retrieve the shared GitHub repo
RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts

# # Set the working directory to where Dagster will look for the application
WORKDIR /opt/dagster/app

# # Copy the project files
COPY --from=builder ./app/requirements.txt /opt/dagster/app
COPY . /opt/dagster/app

# Install all the requirements and then install the package itself
RUN --mount=type=ssh pip install -r requirements.txt
RUN pip install -e .

Answer 2

You probably forgot to activate the virtual environment? In case it is located in /.venv , I suggest to add

# Activate virtual env (i.e. setting VIRTUAL_ENV and extending PATH)
ENV VIRTUAL_ENV=/.venv \
    PATH="/.venv/bin:$PATH"

at the end of the Dockerfile.