Installation of Pyspatialite on Windows

Question

I'm using pyspatialite on ubuntu with no problem—it's perfect. But Windows is giving me a total headache. I've been trying to solve this issue for at least three days.

I'm using Visual Studio 2010 express with mingw (gcc) as the compiler. I encountered and solved a lot of problems: vcvarsall.bat, iconv.h, etc., but I actually don't know exactly what I did. I tried several environment variables for DLLs, .h files (headers), .a files (afaik it's related to mingw), osgeo4w binaries etc. But I couldn't figure it out.

I'm receiving the following error while using MSVC: 'gcc' failed with exit status 1 and while using MinGW: a lot of warning rows at the command prompt related to lex.gml.c, lex.kml.c, lex.geojson.c or something like that.

I'm not a developer. I am just interested in the open source GIS world. Does anyone know of a step-by-step guide for "how to install pyspatialite on windows"?

Answer 1

After a great deal of trial-and-error, google searches, broken keyboards, various forum / chat Q&A sessions, and head injuries - not to mention having to create my own fixes / workarounds for several issues - I've finally succeeded in building and installing a 64-bit version of PySpatiaLite 3.0.1 for Windows x64. Woo-hoo!

Given how difficult it was for me to get to the end-point (several weeks!), I wanted to share this tutorial as an answer to the specific question of, "How do I build and install a 64-bit version of PySpatiaLite on Windows?" Rather than create a new self-answered question thread, I thought it better to just respond to this question about Install PySpatiaLite on Windows, assuming that it would show up for anyone searching for how to do it in a 64-bit build as well. So here goes...

NOTE: The way the Markdown works on this site, I can't fully control the numbering schema of lists; it resets to 1 after each header. So while in the original tutorial I may have numbered something as 30, here I had to note that it shows up as step 5 under Building xyz, etc. What can you do?

---

HOW-TO: INSTALL 64-bit PYSPATIALITE on WINDOWS

In 40 Magnificently Easy-to-Follow Steps - No, really!

---

There are a whole lot of steps here, but only because I've broken them down into their smallest possible components; so don't feel overwhelmed. At a high-level, you're really only doing 5 things:

1. Downloading all the necessary sources
2. Setting up the build environment (you may have already done this anyway)
3. Building and instaling the dependency libraries
4. Building and installing PySpatiaLite
5. Testing your installation

That's it. Not so difficult, really, is it? Oh yes it was!! But hopefully, not so difficult for you following these steps. For many of you, the added detail in these steps will seem like overkill, particularly if you've built from source code before (I hadn't - really!), but I make no assumptions about the skill level or expertise of the audience; so this How-To is written so that a relative beginner (like me!) should be able to follow it.

NOTE: Pay attention to the notes throughout this document, as they often point out some gotchas or things that would not otherwise be obvious at first glance.

NOTE: Always use the latest stable release version where possible, rather than a beta or nightly snapshot. Sometimes, you just have no other choice but to grab that bleeding-edge beta release (or alpha if you know people). For example, you're going to need both the latest stable release of GEOS and a nightly build to workaround a problem.

NOTE: As you travel to some of the links below, you might notice some precompiled binaries of other libraries you're meant to build. I highly recommend you do not use these. You can't be certain of the environment they were built in, and it's better / safer / more certain that things will work properly if you build them in your own environment. The exception to being Gettext.

ME: Stands for "My Environment" as a comparison basis

Requirements:

• A Microsoft Windows x64 OS (Duh! What are we doing here?)
  ME: MS Windows 7 Home x64

• MS Visual Studio Express C++ (MSVC) 2008
  ME: I downloaded and mounted a virtual ISO image rather than use the web-based installer
  NOTE: Choose "vcsetup.exe" (C++) NOT "vcssetup.exe" (C#) - notice the extra 's'? You'd think they'd make it a little more obvious, eh? The full version of MSVC will work equally as well, but like me, I assume free is of great value to you.
  NOTE: Might work with MSVC 2010, but I strongly suggest sticking with same version that Python was compiled on because sometimes it can be a bit prickly, which is why I suggest building with MSVC rather than mingw-w64 or Cygwin.

• MS Software Development Kit (SDK) 7.0 amd64 .NET Framework 3.5
  ME: Also downloaded the ISO image here.
  NOTE: Might work with SDK 7.1 amd64 .NET Framework 4.0 (MSVC 2010), but see note above

• Python 2.7 x64
  ME: Python 2.7.3 x64 (pre-compiled Win x64 install)
  NOTE: Likely will work for version 2.5+, including 3.x

• Latest GNU iconv source code
  ME: libiconv 1.9.2

• Latest Proj.4 source code
  ME: libproj 4.8.0

• Latest GEOS source code
  ME: libgeos 3.3.5 (svn_trunk) + 2012-07-28 nightly snapshot
  NOTE: Also download one of the nightly snapshots from this page; we need to copy a missing directory from it, but we don't want to use it to build the library itself.

• Latest GNU Gettext Runtime BINARY
  ME: Gettext 0.18.1.1-2
  NOTE: We use the binary here instead of building from source code due to a circular dependency with iconv; you can work around it, but this is easier/faster/simpler/less stress- inducing.

• Latest PySpatialLite Package
  ME: PySpatiaLite 3.0.1
  NOTE: As of this writing, 3.0.1 is the latest version available; the specific quirks getting this version to work probably won't all match up with any other version, but much of the steps necessary to compile the dependency libs should still apply.

• Notepad++ or other fully-featured editor app.
  NOTE: Sure, MS NotePad will do in a pinch, but it's going to make this process a whole lot harder. I recommend something a little more robust, and Notepad++ happens to be my favorite.

Setting up the Build Environment:

1. Install MSVC 2008, and run Windows Update (sometimes 2 or more times to get everything), then when no updates remain...

2. Install SDK 7.0, and run Windows Update again (and again, etc., etc., ad nauseum), then when no updates remain...

3. Open a Windows folder (or Windows Explorer) and navigate to 'C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC'

4. Edit 'vcvarsall.bat'

5. Locate the code that look like this at the line numbers indicated:

   Line 19: if not exist "%~dp0bin\amd64\vcvarsamd64.bat" goto missing  
   Line 20: call "%~dp0bin\amd64\vcvarsamd64.bat"  
   

6. The paths in these lines are just plain wrong. Don't ask me why. I guess Microsoft had their heads up their collective arses that day, but if you just remove the '\amd64' and change the file name to 'vcvars64.bat' in each line it should be ok. The end result should look like this:

   Line 19: if not exist "%~dp0bin\vcvars64.bat" goto missing  
   Line 20: call "%~dp0bin\vcvars64.bat"  
   

   NOTE: You can test by opening a command window (START --> RUN --> type 'cmd' --> OK) and running the following command:

   c:\> vcvarsall x64  
   

   You should get a series of output stating that it's 'Setting the environment for using Microsoft Visual Studio 2008 x64 tools'. Go ahead and close this window.

7. Now let's test your SDK 7.0 environment. Click START --> ALL PROGRAMS --> Microsoft Windows SDK 7.0 --> CMD Shell. If all is working properly, you should have a Command line window with yellow text. That indicates a 64-bit environment. Perfect! You can close this window for now.
   NOTE: You might want to create a shortcut to the SDK 7.0 CMD Shell - we'll be using it quite often throughout this process.

8. Decide where you ultimately want to install the dependency libraries. Go ahead and create the folder now if it doesn't already exist. Remember the path you chose, as we'll be referring to it often. For simplicity, we'll call this '\Install' or sometimes 'C:\Install' in example code.

9. Decide where you will keep temporary build files during this install process. This should not be the same location as your '\Install' folder. We'll refer to this as '\Temp' or 'C:\Temp' in example code.

10. Extract the GNU Gettext binary to '\Temp', which should result in a folder called '\Temp\Gettext-0.18.1.1-2' or similar.

11. Copy libintl.dll from '\Temp\Gettext-0.18.1.1-2\bin' to '\Install\bin' (you'll have to create '\bin' yourself), or just copy and paste all the Gettext ('\bin', ''\lib', etc.) sub- folders into '\Install'. Either way works fine.

12. Navigate to '\Install\bin', make a copy of 'libintl.dll' and call it 'intl.dll'. This is necessary for some libraries that look for it under a different name. Now we're ready to proceed with the library builds...

Building the Dependency Libraries:

1. Extract each of the libraries (Proj.4, GEOS, and iconv) into their own folders in '\Temp'. I suggest renaming each folder to something short but descriptive (e.g. '\Temp\Proj4') as typing '\Temp\Proj-4.8.0' (or longer!) over and over again gets tiresome.

Building iconv:

2. Open your SDK CMD Shell, and change directories to your iconv folder (e.g. 'cd c:\temp\iconv').

3. Type either of the following two commands:

   Single-threaded static lib: (recommended)  
   c:\Temp\iconv\> nmake install /f makefile.msvc MFLAGS=-ML PREFIX=c:\Install  
   
   Multi-threaded static lib: (so far it's working for me)  
   c:\Temp\iconv\> nmake install /f makefile.msvc MFLAGS=-MT PREFIX=c:\Install  
   

   Replace 'c:\Install' with your '\Install' path. After a long time of doing some arcane c-style linking and such, the process will eventually complete, and assuming you see no errors that break the build process (e.g. 'cl.exe exited with status 2') - it's relatively safe to ignore the numerous warnings - you're ready to move on.
   NOTE: If you prefer you can build and install the libraries in two separate steps by dropping 'install' right after 'nmake' then rerun the command as above afterwards, but what's the point? If the build portion fails, it won't install anyway.

Building Proj.4:

4. Still in the CMD Shell, change directories to your '\Temp\Proj4' folder (e.g. 'cd c:\temp\proj4').

5. Type the following command:

   c:\Temp\proj4\> nmake install-all /f makefile.vc INSTDIR=c:\Install  
   

   Replace 'c:\Install' with your '\Install' path.
   NOTE: The extension on the makefile here is now '.vc' not '.msvc'. Either seems to be common practice.
   NOTE: Notice that here it's 'nmake install-all ...', which is more common than 'nmake install ...', particularly because 'install' is a built-in nmake command that shouldn't be overridden.
   NOTE: This is the only instance I've seen the use of 'INSTDIR=' rather than 'PREFIX='. It's not wrong per se, just not the standard convention that I've seen used most often.
   Don't worry about any messages such as, 'The system cannot find the file specified. 0 file(s) copied.' The install script is pretty basic and tries to install all .DLL and other files even if you didn't choose to compile them.
   After a pretty short time, the process will complete (again, errors bad, warnings ok), and you can proceed.

Building GEOS:

18. This one is going to take a little extra time and effort. Remember that you downloaded both the latest stable release and a nightly snapshop? If you haven't already, go ahead and extract both to separate folders (e.g. '\Temp\GEOS' and '\Temp\GEOSnap').

19. Copy the folder '\Temp\GEOSnap\src\triangulate' to '\Temp\GEOS\src'. For some reason, as of this writing, '\triangulate' is missing from the latest stable build. That's all we needed from '\GEOSnap'; moving on...

20. Download GEOS\makefile.vc patch to '\Temp\GEOS\makefile.vc'. - Say, 'Thank you.'

21. Download GEOS\src\makefile.vc patch to '\Temp\GEOS\src\makefile.vc'. - You're welcome.
    NOTE: Don't get these two mixed up! If you do, it will go badly for you.
    In addition to implementing a fix from GEOS Trac #574, these two makefiles for MSVC add code to perform basic install and uninstall functions (e.g. 'nmake install-all ...'). Otherwise, you would need to either manually copy over all the files (there's a bunch all over the place!) or use another build method such as CMake, which is currently a bit dodgey.

22. Navigate to '\Temp\GEOS\src', edit 'dirlist.mk', and add the following code at the line number indicated:

    Line 45: triangulate \  
    

    This entry should fall below 'simplify \' and above 'util'. Save and close.

23. Open your CMD Shell, and change directories to your '\Temp\GEOS' folder (e.g. 'cd c:\Temp\GEOS').

24. Type the following command:

    c:\Temp\GEOS\> nmake install-all /f makefile.vc PREFIX=c:\Install  
    

    Replace 'c:\Install' to your '\Install' path. After a long, Long, LONG time, the build process will complete. Assuming no build-breaking errors, you should be notified that GEOS was installed in '\Install'.
    That's all for the dependency libs; now you're (finally) ready to install PySpatiaLite!

Building and Installing PySpatiaLite:

25. Extract PySpatiaLite to '\Temp' in its own folder (e.g. '\Temp\PSL').

26. Navigate to '\Temp\PSL', edit the file called 'setup.py', and modify the following code at the line numbers indicated:

    Line 45: include_dirs = ['c:\Install\include']  
    Line 46: library_dirs = ['c:\Install\libs']  
    Line 47: libraries = ['geos','geos_c','proj','iconv']  
    Line 48: runtime_library_dirs = ['c:\Install\bin']  
    

    Replace the 'c:\Install' portion above with your own '\Install' path, but keep the '\include', '\libs', and '\bin' portions instact. Save and close the file.
    NOTE: You're only adding ",'iconv'" to line 47.

27. Open your CMD Shell and change directories to your PySpatiaLite folder (e.g. 'cd c:\Temp\PSL').

28. Type the following command:

    c:\Temp\PSL\> python setup.py build  
    

    Wait for it to download the spatialite amalgamation, unload the files it requires, begin the build process, and eventually fail.
    Yep. That's right. The build will likely fail (unless Loki has fixed the issue by the time you read this).

29. If and when it does fail, take a look at the console output about 14 lines above the failure message, and you should see buried in that unholy mess of code something like:

    -DVERSION = "3.0.1"  
    

    Make a note of that number; you're going to need it in a couple places going forward.

30. Minimize CMD Shell, navigate to '\Temp\PSL\amalgamation', and edit the file labeled spatialite.c. Don't panic! It's big, it's bad, it's scary, but we're only going to make a small change: a kludgey but effective workaround.

31. Add the following code at the line number indicated:

    Line 54: #define VERSION "3.0.1"  
    

    NOTE: Replace "3.0.1" with the number you saw in step 30 (5 here) above, and make sure you use double quotes (") NOT single quotes ('), or it will fail. Yes, I found this out the hard way, unlike you c gurus out there who know better.
    If you're wondering why we had to wait for it to fail before making this change, it's because the file didn't exist on your system until the amalgamation was downloaded during the first build attempt. So there.

32. Now re-run the command in step 31 above, and the build should complete successfully.

33. Now run the following command:

    c:\Temp\PSL\> python setup.py install  
    

    And whala! Assuming no more bothersome errors, you're finally done with the build/install process. Have a cookie and pat yourself on the back. You can delete all those '\Temp' folders now if you choose. Now for some tests to make sure everything is working properly...

Testing Your Install of PySpatiaLite x64:

34. Open a CMD window and change directories to '..\Python27\Lib\site-packages\pyspatialite\test' (e.g. 'cd c:\python27...\test').

35. Do not start the Python CLI. Type each of these commands in succession:

    c:\python27\...\test\> python dbapi.py  
    c:\python27\...\test\> python dump.py  
    c:\python27\...\test\> python factory.py  
    c:\python27\...\test\> python hooks.py  
    c:\python27\...\test\> python regression.py  
    c:\python27\...\test\> python spatial.py  
    c:\python27\...\test\> python transactions.py  
    c:\python27\...\test\> python types.py  
    c:\python27\...\test\> python userfunctions.py  
    

    After each test, you should receive a report that it ran xx number of tests in x.xxx seconds, and whether the tests passed (OK) or failed. If they all pass, you're golden.
    NOTE: 'spatial.py' may fail as a result of it expecting the SpatiaLite version to be '2.3.1', whereas yours is likely '3.0.1'. Not to worry, a simple edit of the file will fix this and allow you to run the test.

36. Edit 'spatial.py' and change "2.3.1" to the verison you actually installed (remember that number from step 30 (Building PySpatiaLite - step 5) above?) at the line number indicated:

    Line 18: self.assertEqual(row[0], "3.0.1")  
    

    Run the 'spatial.py' test again, and it should work fine now. Not really sure why they'd want to check for the pyspatialite version in this test anyway ... moving on!

    The built-in tests are nice, but how about a more real-world-oriented test? You know, one where you actually create, populate, and poll and manipulate data in a database? Yeah, that sounds just like the ticket!

37. Scroll to the bottom of the Language Bindings: Python section of the online Spatialite Cookbook, and copy the 'spatialite_sample.py' code into a new Python file (e.g. 'psl_sample.py').

38. Before you save, close, and run this test, check for a possible copy/paste error. Make doubly sure the code reads exactly like this at the line numbers indicated:

    Line 28: sql = "SELECT AddGeometryColumn('test_pt', "  
    Line 29: sql += "'geom', 4326, 'POINT', 'XY')"  
    

    I'm guessing somethings getting lost in the translation from HTML code to Python file, because the end space and quote on line 28 kept getting dropped to the beginning of line 29 every time I cut/paste this code, and from more than one source, too. Easy fix, tho.

39. Now save, close, and run it. Do I really have to show you how to do that? C'mon! For a hint, look at step 35 (2 here). ;-)
    Assuming all the tests passed, both the built-in and the cookbook sample, then all indications point to you having a solid 64-bit installation of PySpatiaLite.

40. Celebrate! (needed to round this out to an even 40 ;-) )

I hope this "little" HOW-TO was useful to you. Please feel free to provide feedback / updates / corrections.

Best regards,

-RMWChaos, Your Friendly Neighborhood Evil Overlord

http://www.thedarklands.com

Answer 2

Forget trying to build your own executable!

Just use the existing executable from OSGeo4W.

1. Install OSGeo4W
2. Inside this under app you will find a copy of Python27 and a spatialite site-package
3. Copy the site-package to your own copy of Python
4. Copy all the DLLs in the OsGeo4W/bin directory to a new folder in your copy of Python say binosgeo4w
5. Add the new folder binosgeo4w to your system path

I don't know which DLL is required, but one of them is needed to support the site-package. I was then able to run the test script from http://www.gaia-gis.it/spatialite-2.4.0-4/splite-python.html