Where to place absolute paths in a python project?

Question

I have the following project structure

ts_tools
    /bin
    /docs
    /lib
    /ts_tools
        /data_transfer
            /tests
            data_import.py
            __init__.py
        /data_manipulation
            /tests
            data_smoothing.py
            __init__.py
        __init__.py
    config.yaml.conf
    setup.py
    LICENSE
    README.md
    TODO.md

I would like to import data with the data_import.py file from an external source. I use the config.yaml.conf file to specify the absolute paths of the data with:

root_path:
    windows:
        data_fundamental: C:\\Users\\Malcom\\Data_Fundamental
        data_event: C:\\Users\\Malcom\\Data_Event
    linux:
        data_fundamental: /home/data/data_fundamental
        data_event: /home/data/data_event

The respective paths should be available for all tools in the ts_tools package (i.e. data_import.py and data_smoothing.py). Furthermore, the program should identify the os and choose the path structure accordingly.

I know how to set the paths with the yaml file using

import yaml

with open("config.yaml.conf", "r") as ymlfile:
    cfg = yaml.load(ymlfile)

and I know how to discriminate between the os with

if platform.system().lower() == 'windows':
    ROOT_DATA_PATH = cfg['windows']
else:
    ROOT_DATA_PATH = cfg['linux']

but I don't know where to place these code snippets. I don't think that it is appropriate to use it in the setup.py file. On the other hand I consider it inappropriate to specify a new .py file. What is a good design structure for this problem? Where should a specify absolute file paths? Is my ansatz a step in the right direction?

Thank you in advance.

Answer 1

In this case, you can make it relative to the home directory, so you can have ~/data_fundamental and ~/data_event (Which should be equivalent on both platforms). You can expand this with os.path.expandhome

import os.path

def get_path(s):
    return os.path.normpath(os.path.normcase(
        os.path.expanduser(os.path.expandvars(s))
    ))

# get_path('~/data_fundamental') on Windows:
#    r'c:\users\malcom\data_fundamental'
# get_path('~/data_fundamental') on Linux:
#    r'/home/data/data_fundamental'
# (Assuming your username on Windows is Malcolm
#  and on Linux is data and you haven't changed
#  the default home path)

In any case, having two different setup things might be overly confusing, and you should expand ~ and %VARS% and ${VARS} anyways to make setting it up easier and run as expected.

Your other alternatives include:

• Reading from environment variables
• Writing it in setup.py (You should probably allow some way to change where the config file is as setup.py might put it in a write-protected location)

You could also not have a default at all, and when not given either make a default based on sys.platform() or raise an error message telling the user to set it.

Answer 2

Let's identify two different type of files/data.

• Files/data written by the user or for the user during installation/deploy
• Files/data written by the coder

It can be okay to have absolute paths in files/data defined by the user or generated by the program executing on the user machine. Absolute paths are intrinsically more fragile than relative paths, but it's not that bad in the first case.

In the second case you should never use absolute paths. I see that you are even using two different paths for windows and linux. You don't have to do that and you shouldn't.

In Python you have things such as os.path.expanduser('~') to find the user path, or packages like appdirs. You want to be cross-platform as much as possible, and with Python is almost always possible.