dockerfile copy list of files, when list is taken from a local file

Question

I've got a file containing a list of paths that I need to copy by Dockerfile's COPY command on docker build.

My use case is such: I've got a python requirements.txt file, when inside I'm calling multiple other requirements files inside the project, with -r PATH.

Now, I want to docker COPY all the requirements files alone, run pip install, and then copy the rest of the project (for cache and such). So far i haven't managed to do so with docker COPY command.

No need of help on fetching the paths from the file - I've managed that - just if it is possible to be done - how?

thanks!

Answer 1

Not possible in the sense that the COPY directive allows it out of the box, however if you know the extensions you can use a wildcard for the path such as COPY folder*something*name somewhere/.

For simple requirements.txt fetching that could be:

# but you need to distinguish it somehow
# otherwise it'll overwrite the files and keep the last one
# e.g. rename package/requirements.txt to package-requirements.txt
# and it won't be an issue
COPY */requirements.txt ./
RUN for item in $(ls requirement*);do pip install -r $item;done

But if it gets a bit more complex (as in collecting only specific files, by some custom pattern etc), then, no. However for that case simply use templating either by a simple F-string, format() function or switch to Jinja, create a Dockerfile.tmpl (or whatever you'd want to name a temporary file), then collect the paths, insert into the templated Dockerfile and once ready dump to a file and execute afterwards with docker build.

Example:

# Dockerfile.tmpl
FROM alpine
{{replace}}

# organize files into coherent structures so you don't have too many COPY directives
files = {
    "pattern1": [...],
    "pattern2": [...],
    ...
}
with open("Dockerfile.tmpl", "r") as file:
    text = file.read()

insert = "\n".join([
    f"COPY {' '.join(values)} destination/{key}/"
    for key, values in files.items()
])

with open("Dockerfile", "w") as file:
    file.write(text.replace("{{replace}}", insert))

Answer 2

You might want to do this for example:

FROM ...
ARG files
COPY files

and run with

docker build -build-args items=`${cat list_of_files_to_copy.txt}`