Google Cloud Build using volumes on Dockerfile

Question

I'm trying to build a build pipeline for my application and share a specific folder between steps using volumes.

The problem is because on my first step (unit-tests) I have to install all the libs on the requirements.txt to be able to run my unit tests. And after that I have to build my application running my Dockerfile in other step. I don't want to re-install all the requirements again, so, I thought in copy the requirements already installed and paste them inside the docker build step. Am I able to do that? I followed this thread and tried to replicate to my reality but I still have problems.

Passing files from Google Cloud Container Builder to Docker build task

Here is a sample of what I've done:

My cloudbuild.yaml:

- id: unit-tests
  name: python:3-alpine
  entrypoint: sh
  dir: my-dir
  args:
    - -c
    - |
      apk add --virtual build_dependencies build-base
      apk add --no-cache libstdc++
      pip install --no-cache-dir -r requirements.txt
      apk del build_dependencies
      python -m unittest discover --verbose
      cp -Rv /usr/local/lib/python3.8/site-packages/* /requirements
  volumes:
    - name: 'requirements'
      path: /requirements

- id: docker-build
  name: gcr.io/cloud-builders/docker
  dir: my-dir
  args:
  - build
  - --tag=gcr.io/${PROJECT_ID}/eta:test
  - '.'
  volumes:
    - name: 'requirements'
      path: /requirements
  waitFor: ['unit-tests']

images:
- 'gcr.io/$PROJECT_ID/eta:test'

timeout: 3600s

And here is my Dockerfile:

from python:3-alpine

RUN set -eux; \
    apk add --virtual build_dependencies build-base; \
    apk add --no-cache libstdc++; \
    apk del build_dependencies; 

COPY /requirements/* /usr/local/lib/python3.8/site-packages/
COPY . /app
RUN ["chmod", "+x", "/app/entry.sh"]
WORKDIR /app
ENTRYPOINT [ "/app/entry.sh" ]

I can't locate the /requirements folder inside my dockerfile, here is the error message:

Step #1 - "docker-build": Step 3/7 : COPY /requirements/* /usr/local/lib/python3.8/site-packages/
Step #1 - "docker-build": COPY failed: no source files were specified
Finished Step #1 - "docker-build"
ERROR
ERROR: build step 1 "gcr.io/cloud-builders/docker" failed: step exited with non-zero status: 1

I'm not sure if what I'm doing is the right way.

Answer 1

The Cloud Build (VM) persists /workspace across steps so you may create e.g. /workspace/requirements and use requirements in subsequent steps.

PROJECT=[[YOUR-PROJECT]]
BILLING=[[YOUR-BILLING]]

gcloud projects create ${PROJECT}

gcloud beta billing projects link ${PROJECT} \
--billing-account=${BILLING}

gcloud services enable cloudbuild.googleapis.com \
--project=${PROJECT}

Then:

steps:

  - id: also
    name: busybox
    entrypoint: sh
    dir: x
    args:
      - -c
      - |
        mkdir -p /workspace/requirements
        echo "Hello Freddie" > /workspace/requirements/freddie.txt

  - id: after
    name: gcr.io/cloud-builders/docker
    args:
      - build
      - --tag=gcr.io/${PROJECT_ID}/test
      - .
images:
  - gcr.io/${PROJECT_ID}/test

And Dockerfile:

FROM busybox

WORKDIR /test

COPY /requirements/* .

RUN ls -la

ENTRYPOINT ["more","/test/freddie.txt"]

And:

gcloud builds submit . \
--config=./cloudbuild.yaml \
--project=${PROJECT}

Yields:

BUILD
Starting Step #0 - "also"
Step #0 - "also": Pulling image: busybox
Step #0 - "also": Using default tag: latest
Step #0 - "also": latest: Pulling from library/busybox
Step #0 - "also": Digest: sha256:2ca5e69e
Step #0 - "also": Status: Downloaded newer image for busybox:latest
Step #0 - "also": docker.io/library/busybox:latest
Finished Step #0 - "also"
Starting Step #1 - "after"
Step #1 - "after": Already have image (with digest): gcr.io/cloud-builders/docker
Step #1 - "after": Sending build context to Docker daemon   7.68kB
Step #1 - "after": Step 1/5 : FROM busybox
Step #1 - "after":  ---> 6858809bf669
Step #1 - "after": Step 2/5 : WORKDIR /test
Step #1 - "after":  ---> Running in 7e075adce9eb
Step #1 - "after": Removing intermediate container 7e075adce9eb
Step #1 - "after":  ---> 78f7b2faec16
Step #1 - "after": Step 3/5 : COPY /requirements/* .
Step #1 - "after":  ---> 1ec6a35c1deb
Step #1 - "after": Step 4/5 : RUN ls -la
Step #1 - "after":  ---> Running in 943ddad6434c
Step #1 - "after": total 12
Step #1 - "after": .
Step #1 - "after": ..
Step #1 - "after": freddie.txt
Step #1 - "after": Removing intermediate container 943ddad6434c
Step #1 - "after":  ---> 634072b8aec4
Step #1 - "after": Step 5/5 : ENTRYPOINT ["more","/test/freddie.txt"]
Step #1 - "after":  ---> Running in 13c43b5ca924
Step #1 - "after": Removing intermediate container 13c43b5ca924
Step #1 - "after":  ---> f0ebfe1e8e33
Step #1 - "after": Successfully built f0ebfe1e8e33
Step #1 - "after": Successfully tagged gcr.io/.../test:latest
Finished Step #1 - "after"
PUSH

The problem using volumes is that these won't be in the docker build's context. So, if you'd prefer to use volumes, you'd need to copy the volume's content into /workspace.

Then, when you run the gcr.io/cloud-builders/docker step, /workspace is mounted as the root and so you may access /workspace/requirements as /requirements:

Then:

steps:
  - id: before
    name: busybox
    entrypoint: sh
    dir: x
    args:
      - -c
      - |
        echo "Hello Freddie" > /requirements/freddie.txt
    volumes:
      - name: requirements
        path: /requirements

  - id: during
    name: busybox
    entrypoint: sh
    dir: x
    args:
      - -c
      - |
        more /requirements/freddie.txt
    volumes:
      - name: requirements
        path: /requirements

  - id: fix
    name: busybox
    entrypoint: sh
    args:
      - -c
      - |
        ls -la /workspace
        ls -la /requirements
        mkdir -p /workspace/requirements
        cp -r /requirements /workspace/requirements
        ls -la /workspace/requirements
    volumes:
      - name: requirements
        path: /requirements

  - id: after
    name: gcr.io/cloud-builders/docker
    args:
      - build
      - --tag=gcr.io/${PROJECT_ID}/test
      - .

images:
  - gcr.io/${PROJECT_ID}/test

Yields:

BUILD
Starting Step #0 - "before"
Step #0 - "before": Pulling image: busybox
Step #0 - "before": Using default tag: latest
Step #0 - "before": latest: Pulling from library/busybox
Step #0 - "before": Digest: sha256:2ca5e69e
Step #0 - "before": Status: Downloaded newer image for busybox:latest
Step #0 - "before": docker.io/library/busybox:latest
Finished Step #0 - "before"
Starting Step #1 - "during"
Step #1 - "during": Already have image: busybox
Step #1 - "during": Hello Freddie
Finished Step #1 - "during"
Starting Step #2 - "fix"
Step #2 - "fix": Already have image: busybox
Step #2 - "fix": total 24
Step #2 - "fix": .
Step #2 - "fix": ..
Step #2 - "fix": Dockerfile
Step #2 - "fix": cloudbuild.yaml
Step #2 - "fix": requirements
Finished Step #2 - "fix"
Starting Step #3 - "after"
Step #3 - "after": Already have image (with digest): gcr.io/cloud-builders/docker
Step #3 - "after": Sending build context to Docker daemon  8.192kB
Step #3 - "after": Step 1/5 : FROM busybox
Step #3 - "after":  ---> 6858809bf669
Step #3 - "after": Step 2/5 : WORKDIR /test
Step #3 - "after":  ---> Running in 236aa78f2229
Step #3 - "after": Removing intermediate container 236aa78f2229
Step #3 - "after":  ---> cf05164e8175
Step #3 - "after": Step 3/5 : COPY /requirements/* .
Step #3 - "after":  ---> f5dee710b4d7
Step #3 - "after": Step 4/5 : RUN ls -la
Step #3 - "after":  ---> Running in 8a30309a1536
Step #3 - "after": total 12
Step #3 - "after": .
Step #3 - "after": ..
Step #3 - "after": freddie.txt
Step #3 - "after": Removing intermediate container 8a30309a1536
Step #3 - "after":  ---> 20e2e99c3818
Step #3 - "after": Step 5/5 : ENTRYPOINT ["more","/test/freddie.txt"]
Step #3 - "after":  ---> Running in cc6a57aaa103
Step #3 - "after": Removing intermediate container cc6a57aaa103
Step #3 - "after":  ---> 0a50096e471c
Step #3 - "after": Successfully built 0a50096e471c
Step #3 - "after": Successfully tagged gcr.io/.../test:latest
Finished Step #3 - "after"
PUSH
Pushing gcr.io/.../test
...
DONE