python site-package with same name as SDK package importing regardless of sys.path order

Question

I'm trying to use Google Cloud Shell as a remote development environment for an existing App Engine Standard Python 2.7 application. With the SDK already installed, all of the communication pathways already established (no hassle with cloud auth/app default creds/etc), and full disposable VM available (no need to monkey with virtual env or any other python madness meant to make things work in shared environments), this seems it could be an ideal setup for part-time development/maintenance.

... alas, I am unable to import any of the appengine/google packages.

This appears to be an issue inside of the Google Cloud Shell. So here are the steps to explain it clearly:

Go open a brand-spanking-new empty Google Shell: https://shell.cloud.google.com/

type this to see the path app engine SDK is installed at:

someuser@cloudshell:~$ echo $(gcloud info --format="value(installation.sdk_root)")/platform/google_appengine
/usr/lib/google-cloud-sdk/platform/google_appengine

NOTE that the SDK has a google_appengine/google folder... containing an appengine package:

someuser@cloudshell:~$ ls -la /usr/lib/google-cloud-sdk/platform/google_appengine/google
total 32
drwxr-xr-x  7 root root 4096 Nov 18 02:53 .
drwxr-xr-x  1 root root 4096 Nov 18 02:53 ..
drwxr-xr-x 12 root root 4096 Nov 18 02:53 appengine
-rw-r--r--  1 root root  601 Jan  1  1980 __init__.py
drwxr-xr-x  5 root root 4096 Nov 18 02:53 net
drwxr-xr-x  3 root root 4096 Nov 18 02:53 protobuf
drwxr-xr-x  2 root root 4096 Nov 18 02:54 __pycache__
drwxr-xr-x  3 root root 4096 Nov 18 02:53 pyglib

find the python site-packages location:

someuser@cloudshell:~$ python -c 'import site; print site.getsitepackages()'
['/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages']

NOTE: that one of those directories also has a google folder with different packages:

someuser@cloudshell:~$ ls -la /usr/local/lib/python2.7/dist-packages/google
total 60
drwxr-sr-x 15 root staff 4096 Nov 18 03:12 .
drwxrwsr-x  1 root staff 4096 Nov 18 03:40 ..
drwxr-sr-x  2 root staff 4096 Nov 18 03:11 api
drwxr-sr-x  5 root staff 4096 Nov 18 03:11 api_core
drwxr-sr-x  3 root staff 4096 Nov 18 03:11 _async_resumable_media
drwxr-sr-x  5 root staff 4096 Nov 18 03:11 auth
drwxr-sr-x 29 root staff 4096 Nov 18 03:12 cloud
drwxr-sr-x  3 root staff 4096 Nov 18 03:12 iam
drwxr-sr-x  3 root staff 4096 Nov 18 03:11 logging
drwxr-sr-x  2 root staff 4096 Nov 18 03:11 longrunning
drwxr-sr-x  2 root staff 4096 Nov 18 03:11 oauth2
drwxr-sr-x  6 root staff 4096 Nov 18 03:11 protobuf
drwxr-sr-x  3 root staff 4096 Nov 18 03:11 resumable_media
drwxr-sr-x  3 root staff 4096 Nov 18 03:11 rpc
drwxr-sr-x  2 root staff 4096 Nov 18 03:11 type

ok.. let's start python:

someuser@cloudshell:~$ python

Python 2.7.16 (default, Oct 10 2019, 22:02:15)
[GCC 8.3.0] on linux2

# let's inspect the sys.path

>>> import sys

>>> sys.path
['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages']

# ^^ our SDK path isn't in there.  let's add it to the front of the list in the manner we're supposed to

>>> sys.path.insert(1, '/usr/lib/google-cloud-sdk/platform/google_appengine')

# lets look again

>>> sys.path
['', '/usr/lib/google-cloud-sdk/platform/google_appengine', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-pa
ckages', '/usr/lib/python2.7/dist-packages']

# ok, that should work... RIGHT?


>>> from google.appengine.api import urlfetch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named appengine.api

# NOPE... where's "appengine"???

>>> import google

# ummm....

>>> google.__path__
['/usr/local/lib/python2.7/dist-packages/google']

# ummmmmmmmmm.....

So there are a couple of issues and I don't understand how they relate or how to resolve:

• cloud shell has a google package that conflicts with the google package in the SDK
• why isn't the sys.path being honored anyways?

this has something to do with package namespacing or something... I am not smart enough to figure it out... please help!

Answer 1

The google.appengine module is baked into the first-generation Python (2.7) runtime. It's not available to install via pip, in the second-generation (3.x) runtime, or in Cloud Shell.

The only way to use it is by writing and deploying a first-generation App Engine app.