Looping through sub dirs in large data set, making a new folder with the subdir name, & then hardlinking select files to that new directory

Question

I'm struggling immensely with getting a nested for loop to work for this. The data set I am working with is very large (a little over a million files).

I was looking at a nested for loop but it seems unstable.

count=0
for dir in $(find "$sourceDir" -mindepth 1 -maxdepth 1 -type d)
do
        (
                mkdir -p "$destDir/$dir"
                for file in $(find . -type f)
                do
                        (
                        if [ $((count % 3)) -eq 2 ]
                        then
                                cp -prl "$file" $destDir/$dir
                        fi
                        ((count ++))
                        )
                done
        )
        ((count++))
done

^^ this is only going into the last directory and finding the 3rd file. I need it to enter every directory and find the third file

I've thought of breaking this up into chunks and running several scripts instead of just one to make it more scalable.

Answer 1

I was able to figure out the answer thanks to the commenters!! My input was a folder with 4 sub folders and within each of those 4 subfolders, there are 12 files.

My ideal output was having every 3rd file (starting with three) hardlinked at an external location sorted within their subdirectories... so something like this - subdirA (3rdfile hardlink,6thfile hardlink,9thfile hardlink,12thfile hardlink) subdirB (3rdfile hardlink,6thfile hardlink,...)

... and so on!!

Here is what got it to work:

#!/bin/bash

for d in *;
do
        echo $d
        mkdir Desktop/testjan16/$d

#### loops through each file in the folder and hardlinks every third file (starting w 3) to the appropriate directory 

        for f in `find ./$d -type f | sort | awk 'NR %3 == 0'`; do ln $f Desktop/testjan16/$d; done
done