How to automate concatenating several series of files using bash extended globbing and negation patterns?

Question

thank you so much for any advice and feedback on this matter. This is is my situation:

I have a directory with several hundred files, all that start with foo* and end with *.txt. However, they differ in between the beginning and end with a unique identifier that is "Group#.#" and the files look like so:

foo.Group1.1.txt
foo.Group1.2.txt
foo.Group1.4.txt
foo.Group2.45.txt
.
.
.
foo.Group16.9.txt 

The files begin with Group1 and end at Group 16. They are simple one column txt files, each file has several thousand lines. Each row is a number.

I want to so a series of concatenations with these files in which I concatenate all but those files with the "Group1" and then all the files except "Group1" and "Group2" and then all the files except "Group1", "Group2", and "Group3" and so on, until I am left with just the last Group: "Group16"

In order to do this I use a bash extended globbing expression with a negation syntax to concatenate all files except those that have "Group1" as their ID.

I make a directory "jacks" and output the concatenated file into a txt file within this subdirectory:

cat !(*Group1.*) > jacks/jackknife1.freqs.txt

I can then continue using this command, but adding "Group2" and "Group3" for subsequent concatenations.

cat !(*Group1.*|*Group2.*) > jacks/jackknife2.freqs.txt
cat !(*Group1.*|*Group2.*|*Group3.*) > jacks/jackknife3.freqs.txt

Technically, this works. And 16 Groups isn't too terrible to do this manually. But I am wondering if there is a way, perhaps using loops or bash scripting to automate this process and speed it up?

I would appreciate any advice or leads on this question!

thank you very much, daniela

Answer 1

Some tries around bash globbing

Try using echo before cat !

touch foo.Group{1..3}.{1..5}.txt
ls -l 
total 0
-rw-r--r-- 1 user  user  0 Oct 21 18:37 foo.Group1.1.txt
-rw-r--r-- 1 user  user  0 Oct 21 18:37 foo.Group1.2.txt
-rw-r--r-- 1 user  user  0 Oct 21 18:37 foo.Group1.3.txt
-rw-r--r-- 1 user  user  0 Oct 21 18:37 foo.Group1.4.txt
-rw-r--r-- 1 user  user  0 Oct 21 18:37 foo.Group1.5.txt
-rw-r--r-- 1 user  user  0 Oct 21 18:37 foo.Group2.1.txt
-rw-r--r-- 1 user  user  0 Oct 21 18:37 foo.Group2.2.txt
-rw-r--r-- 1 user  user  0 Oct 21 18:37 foo.Group2.3.txt
-rw-r--r-- 1 user  user  0 Oct 21 18:37 foo.Group2.4.txt
-rw-r--r-- 1 user  user  0 Oct 21 18:37 foo.Group2.5.txt
-rw-r--r-- 1 user  user  0 Oct 21 18:37 foo.Group3.1.txt
-rw-r--r-- 1 user  user  0 Oct 21 18:37 foo.Group3.2.txt
-rw-r--r-- 1 user  user  0 Oct 21 18:37 foo.Group3.3.txt
-rw-r--r-- 1 user  user  0 Oct 21 18:37 foo.Group3.4.txt
-rw-r--r-- 1 user  user  0 Oct 21 18:37 foo.Group3.5.txt

Then

echo !(*Group1.*)
foo.Group2.1.txt foo.Group2.2.txt foo.Group2.3.txt foo.Group2.4.txt foo.Group2.5.txt foo.Group3.1.txt foo.Group3.2.txt foo.Group3.3.txt foo.Group3.4.txt foo.Group3.5.txt

Ok, and

echo !(*Group[23].*)
foo.Group1.1.txt foo.Group1.2.txt foo.Group1.3.txt foo.Group1.4.txt foo.Group1.5.txt

Or

echo !(*Group*(1|3).*)
foo.Group2.1.txt foo.Group2.2.txt foo.Group2.3.txt foo.Group2.4.txt foo.Group2.5.txt

Or even

echo !(*Group*(1|*.3).*)
foo.Group2.1.txt foo.Group2.2.txt foo.Group2.4.txt foo.Group2.5.txt foo.Group3.1.txt foo.Group3.2.txt foo.Group3.4.txt foo.Group3.5.txt

and

echo !(*Group*(1|*.[2-4]).*)
foo.Group2.1.txt foo.Group2.5.txt foo.Group3.1.txt foo.Group3.5.txt

I will let you think about last two sample! ;-)