importing python modules and packages in different sub-directories of the same project

Question

I'd like to figure out the cleanest and preferably self contained way to use my packages in scripts that are in a different directory to the package itself.

The example problem is as follows:

The modules in lib need to both be imported, and run as a script.

My project directory is as below and I'm having two issues:

1. in lib/api.py, I want to read in data_files/key.txt correctly when api.py is called or imported
2. in testing_script.py I want to import and use lib/get_data.py

I can't seem to find a clean way to do this, does this mean my project is structured in a non-pythonic way?

Thanks for the help.

my-project-git
├── LICENSE
├── README.md
├─── my_project
│   ├── data_files
│   │   ├── key.txt
│   │   ├── mappings.csv
│   ├── lib
│   │   ├── __init__.py
│   │   ├── api.py
│   │   └── get_data.py
│   └── test
│       ├── __init__.py
│       └── testing_script.py
├── requirements.txt
└── setup.py

Answer 1

As far as I know, there's isn't a pythonic way to structure your project.

This is what Kenneth Reitz recommended in 2013 and it's how I use it: https://www.kennethreitz.org/essays/repository-structure-and-python.

README.rst
LICENSE
setup.py
requirements.txt
sample/__init__.py
sample/core.py
sample/helpers.py
docs/conf.py
docs/index.rst
tests/test_basic.py
tests/test_advanced.py

Inside sample (my_project in your case) you can separate into categories as you like. E.g. Utils (common functions), Database (read, write), View (user commands), etc. It depends on your project.

As for calling the modules at the same level, you should define them in the __init__ file of the top hierarchy module which is sample in this case.

For example:

__init__ in _my_project

from sample.core import a_Class
from sample.core import a_function
from sample.core import anything

then from /test/test_basic.py you do:

from sample import a_Class
# or import sample

a = a_Class()  # use the class from core.py
# or a = sample.a_Class()

Take a look at the sample module repository: https://github.com/navdeep-G/samplemod