Python code is in one directory, database file is in another. How to open db and process?

Question

I have a directory of db files in folder A. My python code runs from another place.

When I run the following code:

path = 'xxx'                    # path to file directory
filenames = os.listdir(path)    # list the directory file names
#pprint.pprint(filenames)       # print names
newest=max(filenames)
print newest                    # print most recent file name

# would like to open this file and write to it
data=shelve.open(newest, flag="w")

It works up until the last line, then I get an error which says: need "n" or "c" flag to run new db.

Without the flag in the last line eg: data=shelve.open(newest), the file name arrives in the Python code's directory without any data in the db.

I need to be able to put the filename returned by newest in " ", but don't know how.

Answer 1

newest is just the filename (e.g. test.db). Since the current directory (by default the directory from which the script was run) is not the same as the db folder, you need to form a full path. You can do that with os.path.join:

data = shelve.open(os.path.join(path,newest), flag = "w") 

As Geoff Gerrietts points out, max(filenames) returns the filename that comes last in alphabetical order. Perhaps that does give you the file you desire. But if you want the file with the most recent modification time, then you could use

filenames = [os.path.join(path,name) for name in os.listdir(path)]
newest = max(filenames, key = os.path.getmtime)

Note that if you do it this way, then newest will be a full path name, so you would then not need os.path.join in the shelve.open line:

data = shelve.open(newest, flag = "w") 

---

By the way, an alternative to using full path names is to change the current directory:

os.chdir(path)

Although this looks simpler, it can also make your code harder to comprehend, since the reader has to keep track of what the current working directory is. Perhaps this is not hard if you only call os.chdir once, but in a complicated script, calling os.chdir in many places can make the code a bit spaghetti-like.

By using full path names there is no question about what you are doing.

---

If you wish to open each file:

import os
import contextlib

filenames = [os.path.join(path,name) for name in os.listdir(path)]
for filename in filenames:
    with contextlib.closing(shelve.open(filename, flag = "w")) as data:
        # do stuff with data
        # ...
        # data.close() will be called for you when Python leaves this with-block