Jenkins podTemplate using dynamicPVC() is not creating a persistentVolume for my pipeline build

Question

I want to create large EBS volumes (e.g. 1TB) dynamically for each jenkins worker pod on my EKS cluster such that each persistentVolume representing an EBS volume only persists for the lifespan of the pod.

I can't change the size of the nodes on the EKS cluster, so I'm trying to use external EBS volumes via the ebs-csi-driver helm chart.

I can't seem to find a configuration for the podTemplate that enables dynamic provisioning of the persistentVolumeClaim and subsequent EBS persistentVolume for hosting my builds. The only way I'm able to successfully provision and mount EBS persistentVolumes dynamically to my worker pods is by using my own persistentVolumeClaim which I have to reference in the podTemplate and manually deploy using kubectl apply -f pvc.yaml and destroy kubectl delete -f pvc.yaml for each build cycle.

So far I've been able to provision one persistentVolume at a time using the following:

• persistentVolumeClaim:

  apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: ebs-claim
    namespace: jenkins
  spec:
    accessModes:
      - ReadWriteOnce
    storageClassName: ebs-sc
    resources:
      requests:
        storage: 1000Gi 
  

• StorageClass:

  apiVersion: storage.k8s.io/v1
  kind: StorageClass
  metadata:
    name: ebs-sc
  provisioner: ebs.csi.aws.com
  volumeBindingMode: WaitForFirstConsumer
  

• podTemplate:

  apiVersion: v1
  kind: Pod
  spec:
    serviceAccount: jenkins-sa
    securityContext:
      fsGroup: 1000
    containers:
    - name: jenkins-worker
      image: <some-image>
      imagePullPolicy: Always
      command:
      - cat
      tty: true
      securityContext:
        runAsGroup: 1000
        runAsUser: 1000
      volumeMounts:
      - name: build-volume
        mountPath: <some-mount-path>
    volumes:
    - name: build-volume
      persistentVolumeClaim:
        claimName: ebs-claim
  

  • I annotated the "jenkins-sa" serviceAccount with an AWS IAM Role ARN (eks.amazonaws.com/role-arn) with all the AWS IAM policies and trust-relationships needed to allow the jenkins worker Pod to provision an AWS EBS persistentVolume (as described in the AWS ebs-csi-driver docs).

Once the pipeline is started, and the jenkins worker pod is executing a build, I'm able to ssh into it and confirm that the external EBS volume is mouted to the build directory using df. Happy days!

However, I noticed the persistentVolume provisioned by the jenkins worker pod was lingering around after the build finished and the pod destroyed. Apparently this is because the PersistentVolumeClaim needs to be deleted before the persistentVolume it's bound to can be released (see here for details).

After some digging it looks like I need to specify a dynamicPVC() in either the volumes or workspaceVolume spec in the podTemplate yaml. However I'm struggling to find documentation around dynamic PersistentVolumeClaim provisioning beyond the jenkins kubernetes plugin page. Unfortunately when I try to create persistentVolumeClaims dynamically like this it doesn't work and just uses the maximum storage that the node can provision (which is limited to 20Gi).

• Updated podTemplate:

  apiVersion: v1
  kind: Pod
  spec:
    serviceAccount: jenkins-sa
    workspaceVolume:
      dynamicPVC:
        accessModes: ReadWriteOnce
        requestsSize: 800Gi
        storageClassName: ebs-sc
    securityContext:
      fsGroup: 1000
    containers:
    - name: jenkins-worker
      image: <some-image>
      imagePullPolicy: Always
      command:
      - cat
      tty: true
      securityContext:
        runAsGroup: 1000
        runAsUser: 1000
  

I am expecting a new PersistentVolumeClaim and PersistentVolume to be created by the podTemplate when I start my build pipeline, and subsequently destroyed when the pipeline finishes and the pod released.

Answer 1

Many thanks to Saifeddine Rajhi for pointing me in the right direction!

After investigating how the Jenkins helm-chart implements its StatefulSet in the jenkinsci repo and looking at similar issues around provisioning persistentVolumes dynamically for jenkins worker pods, I achieved the behavior I was looking for by specifying dynamicPVC with the workspace workspaceVolume argument as shown here:

pipeline {
    agent {
        kubernetes {
            yaml podTemplate
            workspaceVolume dynamicPVC(accessModes: 'ReadWriteOnce', requestsSize: "1000Gi", , storageClassName: "ebs-sc")
        }
    }
...
}

I then just point my build directory to the default agent workspace directory where the dynamically provisioned EBS volume is mounted to in my container (/home/jenkins/agent).

---

Update (30/11/2023) following comment from Bernard Halas. Thanks for your input! Given that workspaceVolume is not a supported pod spec API element, then it seems the podTemplate approach doesn't support dynamic provisioning of PVCs so is not appropriate for my use-case and certainly explains why that approach wasn't working.

Answer 2

you use k8s 1.23+, then this is possible for StatefulSets, i.e if you can change pods by StatefulSets.

The PersistentVolumeClaim retention feature configures the volume retention behavior that applies when the StatefulSet is deleted with the field whenDeleted

So when PersistentVolumeClaims (PVCs) generated from the StatefulSet spec template will be deleted automatically when the StatefulSet is deleted or pods in the StatefulSet are scaled down.

kind: StatefulSet
...
spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Delete
    whenScaled: Delete
...

Also, add the reclaimPolicy: Delete field in the StorageClass resource which determines what happens to the Persistent Volumes (PVs) associated with that StorageClass when they are dynamically provisioned and later released.

As a result:

• When a PV associated with this StorageClass is dynamically provisioned and later released, the PV and the underlying storage resources are automatically deleted.
• This means that when the associated PVC is deleted, both the PV and the underlying storage are automatically cleaned up.