How to execute a prefect Flow on a docker image?

Question

My goal:

I have a built docker image and want to run all my Flows on that image.

Currently:

I have the following task which is running on a Local Dask Executor. The server on which the agent is running is a different python environment from the one needed to execute my_task - hence the need to run inside a pre-build image.

My question is: How do I run this Flow on a Dask Executor such that it runs on the docker image I provide (as environment)?

import prefect
from prefect import task, Flow
from prefect.engine.executors import LocalDaskExecutor
from prefect.environments import LocalEnvironment


@task
def hello_task():
    logger = prefect.context.get("logger")
    logger.info("Hello, Docker!")


with Flow("My Flow") as flow:
    results = hello_task()

flow.environment = LocalEnvironment(
    labels=[], executor=LocalDaskExecutor(scheduler="threads", num_workers=2),
)

I thought that I need to start the server and the agent on that docker image first (as discussed here), but I guess there can be a way to simply run the Flow on a provided image.

Update 1

Following this tutorial, I tried the following:

import prefect
from prefect import task, Flow
from prefect.engine.executors import LocalDaskExecutor
from prefect.environments import LocalEnvironment
from prefect.environments.storage import Docker


@task
def hello_task():
    logger = prefect.context.get("logger")
    logger.info("Hello, Docker!")


with Flow("My Flow") as flow:
    results = hello_task()

flow.storage = Docker(registry_url='registry.gitlab.com/my-repo/image-library')
flow.environment = LocalEnvironment(
    labels=[], executor=LocalDaskExecutor(scheduler="threads", num_workers=2),
)

flow.register(project_name="testing")

But this created an image which it then uploaded to the registry_url provided. Afterwards when I tried to run the registered task, it pulled the newly created image and the task is stuck in status Submitted for execution for minutes now.

I don't understand why it pushed an image and then pulled it? Instead I already have an image build on this registry, I'd like to specify an image which should be used for task execution.

Answer 1

The way i ended up achieve this is as follows:

1. Run prefect server start on the server (i.e. not inside docker). Apparently docker-compose in docker is not a good idea.
2. Run prefect agent start inside the docker image
3. Make sure the flows are accessible by the docker image (i.e. by mounting a shared volume between the image and the server for example)

You can see the source of my answer here.