gsutil error "ImportError: cannot import name constants" i can't fix

Question

I'm trying to use the gsutil command to download files from gs bucket. This is being down from a Google Cloud VM Instance and it was working for me until friday when it stopped working, without any changes on my side.

The OS is CentOS 6 for whatever reason, so default python is 2.6. When i started my work on this box i was able to use the env variable CLOUDSDK_PYTHON to specify the python 2.7 path, and everything worked for a few weeks. Then out of the blue it stopped working with some python error from deep inside gsutil. I found a backup version on the box and have been using that, until friday when it stopped working.

With CLOUDSDK_PYTHON set, this is the error i'm getting

[daily-gateway-node bin]$ gsutil -version Traceback (most recent call last): File "/usr/local/bin/gsutil", line 7, in from gslib.main import main File "/usr/local/lib/python2.7/site-packages/gslib/main.py", line 39, in import boto File "/usr/local/lib/python2.7/site-packages/boto/init.py", line 1216, in boto.plugin.load_plugins(config) File "/usr/local/lib/python2.7/site-packages/boto/plugin.py", line 93, in load_plugins _import_module(file) File "/usr/local/lib/python2.7/site-packages/boto/plugin.py", line 75, in _import_module return imp.load_module(name, file, filename, data) File "/usr/lib/python2.6/site-packages/google_compute_engine/boto/boto_config.py", line 30, in from google_compute_engine import constants ImportError: cannot import name constant

Any help would be greatly appreciated. Ultimately what i'm trying to do is get data off BigQuery, but unfortunately Google only allows us to import from local to Big Query but not export from Big Query to local, so i'm stuck using this gsutil library that has contstantly given me problems. I'm able to run bq commands just fine.

So far I've googled this problem many times and tried various solutions ranging from setting CLOUDSDK_PYTHON to running gsutil with something like "/usr/local/bin/python2.7 /usr/bin/gsutil -l" but everything gives the above error. It appears python 2.7 is getting used, until the last call when it goes back to 2.6 dir. I've tried re-installing google cloud sdk with both yum and pip but both say they are up to date.

Answer 1

I gave up trying to get python 2.6 working with gsutil, as nothing i found was working, but i did manage to get things working.

My solution was to spin up another Google Cloud Instance with python > 2.7, which was ubuntu 16.04, and run gsutil from that box, then use gcloud compute ssh to execute the gsutil command remotely and then use gcloud compute scp to copy to the CentOS 6 box.

These are run from the CentOS 6 box to copy all files within a directory:

gcloud compute ssh [GC_Host_For_gsutil] --zone=us-west1-a -- "gsutil -m cp -r \ [GS_BUCKET_WITH_NO_SLASH] [NFS_DIR_WITH_SLASH];"

gcloud compute scp --zone=us-west1-a --recurse \
[GC_Host_For_gsutil]:[NFS_DIR_WITH_SLASH]* [final_destination_no_slash]