Python 2.7 Virtualenv Path issues on OSX 10.5

Question

Problem Discription

The issues I'm having when I try to load a Django website that's using mod_wsgi, python 2.7 and virtualenv, is that I get a 500 error when I go to the site in my browser. When I look at the error that's created in my apache log I get the following...

[Tue Oct 11 09:32:30 2011] [error] [client 206.130.131.226] mod_wsgi (pid=69270): Exception occurred processing WSGI script '/Volumes/Ontario/Sites/emails/config/apache/dev.emails.wenatcheeworld.com.wsgi'.
[Tue Oct 11 09:32:30 2011] [error] [client 206.130.131.226] Traceback (most recent call last):
[Tue Oct 11 09:32:30 2011] [error] [client 206.130.131.226]   File "/Library/Python/2.5/site-packages/Django-1.2.1-py2.5.egg/django/core/handlers/wsgi.py", line 230, in __call__
[Tue Oct 11 09:32:30 2011] [error] [client 206.130.131.226]     self.load_middleware()
[Tue Oct 11 09:32:30 2011] [error] [client 206.130.131.226]   File "/Library/Python/2.5/site-packages/Django-1.2.1-py2.5.egg/django/core/handlers/base.py", line 33, in load_middleware
[Tue Oct 11 09:32:30 2011] [error] [client 206.130.131.226]     for middleware_path in settings.MIDDLEWARE_CLASSES:
[Tue Oct 11 09:32:30 2011] [error] [client 206.130.131.226]   File "/Library/Python/2.5/site-packages/Django-1.2.1-py2.5.egg/django/utils/functional.py", line 276, in __getattr__
[Tue Oct 11 09:32:30 2011] [error] [client 206.130.131.226]     self._setup()
[Tue Oct 11 09:32:30 2011] [error] [client 206.130.131.226]   File "/Library/Python/2.5/site-packages/Django-1.2.1-py2.5.egg/django/conf/__init__.py", line 40, in _setup
[Tue Oct 11 09:32:30 2011] [error] [client 206.130.131.226]     self._wrapped = Settings(settings_module)
[Tue Oct 11 09:32:30 2011] [error] [client 206.130.131.226]   File "/Library/Python/2.5/site-packages/Django-1.2.1-py2.5.egg/django/conf/__init__.py", line 75, in __init__
[Tue Oct 11 09:32:30 2011] [error] [client 206.130.131.226]     raise ImportError("Could not import settings '%s' (Is it on sys.path? Does it have syntax errors?): %s" % (self.SETTINGS_MODULE, e))
[Tue Oct 11 09:32:30 2011] [error] [client 206.130.131.226] ImportError: Could not import settings 'config.settings' (Is it on sys.path? Does it have syntax errors?): No module named config.settings

The traceback doesn't make sense because the files that are getting loaded are from the global python 2.5 library.

Virtualenv Setup Information

I installed python 2.7 on an OS X Server that's running OS X 10.5.8. I installed python using the Python 2.7.2 Mac OS X 32-bit i386/PPC Installer. The server currently has virtualenv 1.6.4 installed.

I setup my virtualenv using for a new website using the following

$ cd /usr/local/virtualenvs
$ virtualenv --no-site-packages -p /usr/local/bin/python2.7 dev.emails

I then install all the nessary python packages. When I run pip freeze this is my output

$ pip freeze
Django==1.3
PIL==1.1.7
South==0.7.3
amqplib==1.0.2
anyjson==0.3.1
boto==2.0
celery==2.2.7
django-celery==2.2.4
django-compressor==0.9.2
django-grappelli==2.3.4
django-picklefield==0.1.9
feedparser==5.0.1
html2text==3.02
kombu==1.4.1
lxml==2.2.2
psycopg2==2.4.2
pyparsing==1.5.6
python-dateutil==1.5
python-sendgrid==0.2.0dev
redis==2.4.9
requests==0.6.2
wsgiref==0.1.2

Trouble Shooting Steps

I created a script called path_test.py in order to test my virtualenv path...

import os
import sys

# Activate Python Virtual Enviroment
activate_this = '/usr/local/virtualenvs/dev.emails/bin/activate_this.py'
execfile(activate_this, dict(__file__=activate_this))

# start django
import django.core.handlers.wsgi
os.environ['DJANGO_SETTINGS_MODULE'] = 'config.settings'

for p in sys.path:
        print p

application = django.core.handlers.wsgi.WSGIHandler()

When I run the script using python path_test.py I get the following output:

/usr/local/virtualenvs/dev.emails/lib/python2.5/site-packages
/Volumes/Ontario/Sites/emails/bin
/Library/Python/2.5/site-packages/MySQL_python-1.2.2-py2.5-macosx-10.5-i386.egg
/Library/Python/2.5/site-packages/paramiko-1.7.4-py2.5.egg
/Library/Python/2.5/site-packages/simplejson-2.0.7-py2.5-macosx-10.5-i386.egg
/Library/Python/2.5/site-packages/python_ldap-2.3.7-py2.5-macosx-10.5-i386.egg
/Library/Python/2.5/site-packages/lxml-2.2.2-py2.5-macosx-10.5-i386.egg
/Library/Python/2.5/site-packages/pytz-2010h-py2.5.egg
/Library/Python/2.5/site-packages/python_openid-2.2.4-py2.5.egg
/Library/Python/2.5/site-packages/Markdown-2.0.3-py2.5.egg
/Library/Python/2.5/site-packages/django_authopenid-1.0.1-py2.5.egg
/Library/Python/2.5/site-packages/Django-1.2.1-py2.5.egg
/Library/Python/2.5/site-packages/django_profiles-0.2-py2.5.egg
/Library/Python/2.5/site-packages/django_openid_consumer-0.1.1-py2.5.egg
/Library/Python/2.5/site-packages/setuptools-0.6c11-py2.5.egg
/Library/Python/2.5/site-packages/ipython-0.10-py2.5.egg
/Library/Python/2.5/site-packages/httplib2-0.6.0-py2.5.egg
/Library/Python/2.5/site-packages/guess_language-0.2-py2.5.egg
/Library/Python/2.5/site-packages/django_ratings-0.3.4-py2.5.egg
/Library/Python/2.5/site-packages/pip-0.8.1-py2.5.egg
/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python25.zip
/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/plat-darwin
/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/plat-mac
/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/plat-mac/lib-scriptpackages
/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python
/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-tk
/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/lib-dynload
/Library/Python/2.5/site-packages
/Library/Python/2.5/site-packages/PIL
/System/Library/Frameworks/Python.framework/Versions/2.5/Extras/lib/python/PyObjC

If I activate the virtualenv first using source /path/to/virtualenv/bin/activate and then run python path_test.py I get the following output:

/usr/local/virtualenvs/dev.emails/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg
/usr/local/virtualenvs/dev.emails/lib/python2.7/site-packages/pip-1.0.2-py2.7.egg
/usr/local/virtualenvs/dev.emails/lib/python2.7/site-packages/lxml-2.2.2-py2.7-macosx-10.3-intel.egg
/usr/local/virtualenvs/dev.emails/lib/python2.7/site-packages
/usr/local/virtualenvs/dev.emails/lib/python2.7/site-packages/PIL
/Volumes/Ontario/Sites/emails/bin
/Volumes/Ontario/local/virtualenvs/dev.emails/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg
/Volumes/Ontario/local/virtualenvs/dev.emails/lib/python2.7/site-packages/pip-1.0.2-py2.7.egg
/Volumes/Ontario/local/virtualenvs/dev.emails/lib/python2.7/site-packages/lxml-2.2.2-py2.7-macosx-10.3-intel.egg
/Volumes/Ontario/local/virtualenvs/dev.emails/lib/python27.zip
/Volumes/Ontario/local/virtualenvs/dev.emails/lib/python2.7
/Volumes/Ontario/local/virtualenvs/dev.emails/lib/python2.7/plat-darwin
/Volumes/Ontario/local/virtualenvs/dev.emails/lib/python2.7/plat-mac
/Volumes/Ontario/local/virtualenvs/dev.emails/lib/python2.7/plat-mac/lib-scriptpackages
/Volumes/Ontario/local/virtualenvs/dev.emails/lib/python2.7/lib-tk
/Volumes/Ontario/local/virtualenvs/dev.emails/lib/python2.7/lib-old
/Volumes/Ontario/local/virtualenvs/dev.emails/lib/python2.7/lib-dynload
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac
/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages
/Volumes/Ontario/local/virtualenvs/dev.emails/lib/python2.7/site-packages
/Volumes/Ontario/local/virtualenvs/dev.emails/lib/python2.7/site-packages/PIL
/Volumes/Ontario/Sites/emails

At this point I'm pretty sure the problem is that activate_this.py in the virtualenv isn't setting up the correct sys.path.

Answer 1

As I pointed out in my reply to my own question and as others have replied it's not possible to use different version of python with mod_wsgi. My solution was to install and setup gunicorn for my specific site that needed a different version of python.

First I installed gunicorn...

$ /usr/local/virtualenvs/my.examplesite/bin/activate
$ pip install gunicorn

Then I setup a launchd.plist to run gunicorn in /Library/LaunchDaemons/my.examplesite.com.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
        <key>Label</key><string>my.examplesite.com</string>
        <key>EnvironmentVariables</key>
        <dict>
            <key>DJANGO_SETTINGS_MODULE</key><string>config.settings</string>
        </dict>
        <key>UserName</key><string>_www</string>
        <key>ProgramArguments</key>
        <array>
            <string>/usr/local/virtualenvs/my.examplesite/bin/gunicorn_django</string>
            <string>--bind=127.0.0.1:8001</string>
        </array>
        <key>RunAtLoad</key><true/>
        <key>StandardErrorPath</key><string>/var/log/gunicorn/my.examplesite.com.error.log</string>
        <key>StandardOutPath</key><string>/var/log/gunicorn/my.examplesite.com.access.log</string>
    </dict>
</plist>

Next I setup nginx to server media and proxy to my gunicorn server:

server {

    listen       80;
    server_name  my.examplesite.com;
    access_log   /var/log/nginx/my.examplesite.com.access.log;
    error_log    /var/log/nginx/my.examplesite.com.error.log;

    location = /favicon.ico {
        return  404;
    }

    location  /static/ {
        root  /path/to/site/root/;
    }

    location  /media/ {
        root  /path/to/site/root/;
    }

    location  / {
        proxy_pass            http://127.0.0.1:8001/;
        proxy_redirect        off;
        proxy_set_header      Host             $host;
        proxy_set_header      X-Real-IP        $remote_addr;
        proxy_set_header      X-Forwarded-For  $proxy_add_x_forwarded_for;
        client_max_body_size  10m;
    }

}

Answer 2

It looks like mod_wsgi is defaulting to Python 2.5. Did you compile it correctly? Have a look at http://code.google.com/p/modwsgi/wiki/InstallationIssues#Multiple%5FPython%5FVersions.

From the page on virtualenv and mod_wsgi[1]:

Note that the version of Python from which this baseline environment is created must be the same version of Python that mod_wsgi was compiled for. It is not possible to mix environments based on different major/minor versions of Python.

[1] http://code.google.com/p/modwsgi/wiki/VirtualEnvironments

Answer 3

You've got a couple of problems.

First, activate_this.py is working fine. The problem is with your setup. You're trying to activate the virtualenv, but you're using the system's default 2.5 python and not the virtualenv's 2.7 python. That is why you are seeing the system site-packages; ''--no-site-packages" only applies to the python binary created by virtualenv. That is also why it is adding "/usr/local/virtualenvs/dev.emails/lib/python2.5/site-packages". The activate_this.py script uses the version of python executing the script to determine the site-packages directory to use:

site_packages = os.path.join(base, 'lib', 'python%s' % sys.version[:3], 'site-packages')

When you activate the virtualenv, it uses the correct 2.7 binary.

Second, your version of mod_wsgi was built against the system's 2.5 python, so it is encountering the same problem you see when the virtualenv isn't activated. The "activate_this.py" script actually does nothing to enforce the "--no-site-packages" option, it relies on the python binary created by virtualenv, so mod_wsgi will never enforce "--no-site-packages". Which is why you see it fall back to the version of Django installed on the system when it doesn't find a 2.5 site-packages directory in the virtualenv.

Third, Apache runs in 64-bit on 10.5 and you've installed a python binary that is 32-bit only, so even if you rebuilt mod_wsgi against your 2.7 version, Apache would crash when trying to load it. And of course all of your 2.5 sites would have the same problem your 2.7 ones do now. Your best bet is to run your 2.7 sites behind something like gunicorn.

Answer 4

If you have built mod_wsgi from source code against Python 2.7, but find it picking up Python 2.5 framework (and not crashing for some reason), try using the:

--disable-framework

to 'configure' when building mod_wsgi from source code with Python 2.7.

If you didn't even rebuild mod_wsgi against Python 2.7, then do that first.