Recursively expand and search pattern of specific subdirectories

Question

I'm looking for an option to search specific subdirectories in python.

For instance a directory structure like this:

some_files/
     common/
     2009/
     2010/
     2011/
     ...

I only want to search for in the subdirectories that start with a 2, so that it must be something like 'some_files/2*'. I think it must be possible using glob.glob and os.walk(), but I can't get it to work.

Now i use:

files = [os.path.join(dirpath, f)
                for dirpath, dirnames, files in os.walk(d)
                for f in files if f.endswith(ext)]

but this doesn't fit the specific needs.

Can someone help me out, would be much appreciated!

Answer 1

You can use glob with dirpath to find matching directories:

from glob import iglob
import os

files = []
ext = "py"
for dirpath, dirnames, file in os.walk(path):
    match = next(iglob(os.path.join(dirpath, "2*")),"")
    if match:
        files.extend(iglob(os.path.join(match,"*.{}".format(ext))))
print(files)

Or if you really want a list comp:

files = [f for dirpath, dirnames, file in os.walk(path) for f in
         iglob(os.path.join(next(iglob(os.path.join(dirpath, "2*")),
                                 '\\\\'), "*.{}".format(ext)))]
print(files)

Answer 2

I would do it like this using pathlib (which is now part of the Python3 std lib):

from pathlib import Path

for subpath in Path().glob("2*):
    for file in subpath.glob("*.ext"):
        # ...

Update: pathlib is also available for Python 2.x (it was back-ported and published to the Python Package Index). Simply:

$ pip install pathlib