Execute a Big query in cloud buils step with multiple source files

Question

I have my cloud build file like below. steps:

• name: gcr.io/cloud-builders/gcloud args:
  • config
  • set
  • project
  • '${_PROJECT_ID}'
• name: gcr.io/cloud-builders/gcloud args:
  • query
  • '--use_legacy_sql=false'
  • '--project_id=${_PROJECT_ID}'
  • '--nouse_legacy_sql'
  • '--dataset=$(_PROJECT_ID}:${_DATASET_ID}'
  • '--source=${_SQL_FILE}' entrypoint: bq

It runs for only one sql file at a time. How to pas multiple source files at a time?

I have tried giving only one source file.

Answer 1

Add additional steps to run the query for multiple SQL files. You can continue this to include more SQL files by adding more steps with different variable names. As mentioned in this document1 & document2

A build step specifies an action that you want Cloud Build to perform. By default, Cloud Build executes all steps of a build serially on the same machine. If you have steps that can run concurrently, use the waitFor option. You can include up to 300 build steps in your config file. The following snippet shows a build config with two steps that runs serially:

steps:
- name: foo
- name: bar

The example below uses the id field to identify certain build steps. The values from id are used in waitFor to define build step order:

• First, fetch-resources step uses gsutil to copy the local resources from Cloud Storage. Concurrently, go generates, tests, and installs the source code.

• Then, the docker build step builds the image after all other steps are complete.

    - name: 'gcr.io/cloud-builders/go'
      args: ['generate']
    - name: 'gcr.io/cloud-builders/go'
      args: ['test', './...']
    - name: 'gcr.io/cloud-builders/go'
      args: ['install', 'mytarget']
      id: 'go-install'   
    - name: 'gcr.io/cloud-builders/gsutil'
      args: ['cp', '-r', './somefiles', 'gs://my-resource-bucket/somefiles']
      waitFor: ['-']  # The '-' indicates that this step begins immediately.
      id: 'fetch-resources'    
    - name: 'gcr.io/cloud-builders/docker'
      args: ['build', '-t', 'gcr.io/$PROJECT_ID/mytarget', '.']
      waitFor: ['go-install', 'fetch-resources'] 
    images: ['gcr.io/$PROJECT_ID/mytarget'] ```