Mirror only files having specific string in file path

Question

I'm trying to mirror only those branches of a directory tree that contain a specific directory name somewhere within the branch. I've spent several hours trying different things to no avail.

A remote FTP site has a directory structure like this:

image_db
  movies
    v2
      20131225
        xyz
          xyz.jpg
      20131231
        abc
          abc.jpg
      AllPhotos   <-- this is what I want to mirror
        xyz
          xyz.jpg
        abc
          abc.jpg
    v4
      (similar structure to 'v2' above, contains 'AllPhotos')
    ...
  tv_shows
    (similar structure to 'movies', contains 'AllPhotos')
  other
    (different paths, some of which contain 'AllPhotos')
  ...

I am trying to create a local mirror of only the 'AllPhotos' directories, with their parent paths intact.

I've tried variations of this:

lftp -e 'mirror --only-newer --use-pget-n=4 --verbose -X /* -I AllPhotos/ /image_db/ /var/www/html/mir_images' -u username,password ftp.example.com

...where the "-X /*" excludes all directories and "-I AllPhotos/" includes only AllPhotos. This doesn't work, lftp just copies everything.

I also tried variations of this:

lftp -e 'glob -d -- mirror --only-newer --use-pget-n=4 --verbose /image_db/*/*/AllPhotos/ /var/www/html/mir_images' -u username,password ftp.example.com

...and lftp crunches away at the remote directory structure without actually creating anything on my side.

Basically, I want to mirror only those files that have the string 'AllPhotos' somewhere in the full directory path.

Update 1:

If I can do this with wget, rsync, ftpcopy or some other utility besides lftp, I welcome suggestions for alternatives.

Trying wget didn't work for me either:

wget -m -q -I /image_db/*/*/AllPhotos ftp://username:[email protected]/image_db

...it just gets the whole directory structure, even though the wget documentation says that wildcards are permitted in -I paths.

Update 2:

After further investigation, I am coming to the conclusion that I should probably write my own mirroring utility, although I still suspect I am approaching lftp the wrong way, and that there's a way to make it mirror only files that have a specific string in the absolute path.

Answer 1

One solution :

curl -s 'ftp://domain.tld/path' |
    awk '/^d.*regex/{print $NF}' |
    xargs wget -m ftp://domain.tld/path/

Or using lftp :

lftp -e 'ls; quit' 'ftp://domain.tld/path' |
    awk '/^d.*regex/{print $NF}' |
    xargs -I% lftp -e "mirror -e %; quit" ftp://domain.tld/path/