How to check if file exists in Google Cloud Storage with the gcloud bash command?

Question

I need to check if a file exists in a gitlab deployment pipeline. How to do it efficiently and reliably?

Answer 1

This answer evolved from the answer of Mazlum Tosun. Because I think it is a substantial improvement with less lines and no global settings switching it needs to be a separate answer.

Ideally the answer would be something like this

- gsutil stat $BUCKET_PATH
- if [ $? -eq 0 ]; then  
  ... # do if file exists
  else
  ... # do if file does not exists
  fi

$? stores the exit_status of the previous command. 0 if success. This works fine in a local console. The problem with Gitlab will be that if the file does not exists, then "gsutil stat $BUCKET_PATH" will produce a non-zero exit code and the whole pipeline will stop at that line with an error. We need to catch the error, while still storing the exit code.

We will use the or operator || to suppress the error. FILE_EXISTS=false will only be executed if gsutil stat fails.

- gsutil stat $BUCKET_PATH || FILE_EXISTS=false
- if [ "$FILE_EXISTS" = false ]; then 
  ... # do stuff if file does not exist
  else
  ... # do stuff if file exists
  fi

Also we can use the -q flag to let the command stats be silent if that is desired.

- gsutil -q stat $BUCKET_PATH || FILE_EXISTS=false

Answer 2

Use gsutil ls gs://bucket/object-name and check the return value for 0.

If the object does not exist, the return value is 1.

Answer 3

You can add the following Shell script in a Gitlab job :

#!/usr/bin/env bash

set -o pipefail
set -u

gsutil -q stat gs://your_bucket/folder/your_file.csv

PATH_EXIST=$?
if [ ${PATH_EXIST} -eq 0 ]; then
  echo "Exist"
else
  echo "Not Exist"
fi

I used gcloud cli and gsutil with stat command with -q option.

In this case, if the file exists the command returns 0 otherwise 1.