How to run parallel stages in isolated pods for a declarative jenkins pipeline

Question

I'm trying to run some end-to-end tests in parallel and on DIFFERENT kubernetes pods on a declarative jenkins pipeline, jenkins however seems to attempt running the parallel stages on the SAME kubernetes pod. This leads to database deadlocks since both processes try inserting/truncating/updating/querying the same tables. Is there a way I can spin up a different pod for each of the parallel stages?

The kubernetes-plugin config:

  agent {
  kubernetes {
    label 'my-label'
    defaultContainer 'jnlp'
    yaml """
apiVersion: v1
kind: Pod
metadata:
  name: dind
spec:
  containers:
    - name: < default container >
      image: < image >
      securityContext:
          privileged: true
          fsGroup: 1000
      command:
      - cat
      tty: true
      volumeMounts:
        - name: jenkins-bundle-gems
          mountPath: /usr/local/bundle


    - name: <tests-container-name>
      image: < image > 
      securityContext:
          privileged: true
          fsGroup: 1000
      volumeMounts:
        - name: jenkins-bundle-gems
          mountPath: /usr/local/bundle
      command:
      - cat
      tty: true
"""
 }
   }

The parallel stage:

      stage('Test'){
        parallel {
          stage("Branch 1") {
              steps {
                container('<tests-container-name>') {
                  sh "jenkins/scripts/initdb.sh"
                  sh 'bundle exec rspec --exclude-pattern "spec/features/*_spec.rb" spec'
                }
              }
            }
            stage("Branch 2") {
              steps {
                container('<tests-container-name>') {
                  sh "jenkins/scripts/initdb.sh"
                  sh "bundle exec rspec `jenkins/scripts/split_features.sh 0`"
                }
              }
            }
        }
      }

EXPECTATION: I would like for jenkins to spin up two different pods for each of the parallel stages. This would allow me to use different databases for each of the tests.

ACTUAL RESULT: Jenkins runs both stages concurrently on the same pod.

Answer 1

You have__________: stage > parallel > stage > steps.

You need to have also: stage > parallel > stage > agent.

Do not repeat pod definition twice, it is recommended to put the pod definition in a separated file and refer to it using yamlFile instead of yaml :

  stage('Test'){
    parallel {
      stage("Branch 1") {
          agent {
             kubernetes {
               defaultContainer 'jnlp'
               yamlFile 'Jenkins.pod.yaml'
             }
          }
          steps {
            container('<tests-container-name>') {
              sh "jenkins/scripts/initdb.sh"
              sh 'bundle exec rspec --exclude-pattern "spec/features/*_spec.rb" spec'
            }
          }
        }
        stage("Branch 2") {
          agent {
             kubernetes {
               defaultContainer 'jnlp'
               yamlFile 'jenkins.pod.yaml'
             }
          }
          steps {
            container('<tests-container-name>') {
              sh "jenkins/scripts/initdb.sh"
              sh "bundle exec rspec `jenkins/scripts/split_features.sh 0`"
            }
          }
        }
    }
  }

Hint

if blueocean is one of your plugins, it will help you to draw your pipeline under http://HOST/blue/organizations/jenkins/pipeline-editor/, then you can copy the Jenkinsfile code by typing [Cmd + s]

Answer 2

Try something like this

stage("Run additional parallel tests") {
        parallel( 
            "parallel stage 1": {
                [INSERT YOUR CODE HERE]
            },

            "parallel stage 2": {
                [INSERT YOUR CODE HERE]
            }
        )
    }
}

Answer 3

You can set agent {} for each parallel stage to launch a pod per stage.