Perform command from command line inside directories from glob in bash shell script

Question

In a bash shell script do-for.sh I want to perform a command inside all directories named in a glob using bash. This has been answered oodles of times, but I want to provide the command itself on the command line. In other words assuming I have the directories:

• foo
• bar

I want to enter

do-for * pwd

and have bash print the working directory inside foo and then inside bar.

From reading the umpteen answers on the web, I thought I could do this:

for dir in $1; do
  pushd ${dir}
  $2 $3 $4 $5 $6 $7 $8 $9
  popd
done

Apparently though the glob * gets expanded into the other command line arguments variable! So the first time through the loop, for $2 $3 $4 $5 $6 $7 $8 $9 I expected foo pwd but instead it appears I get foo bar!

How can I keep the glob on the command line from being expanded into the other parameters? Or is there a better way to approach this?

To make this clearer, here is how I want to use the batch file. (This works fine on the Windows batch file version, by the way.)

./do-for.sh repo-* git commit -a -m "Added new files."

Answer 1

I will assume you are open to your users having to provide some kind of separator, like so

./do-for.sh repo-* -- git commit -a -m "Added new files."

Your script could do something like (this is just to explain the concept, I have not tested the actual code) :

CURRENT_DIR="$PWD"

declare -a FILES=()

for ARG in "$@"
do
  [[ "$ARG" != "--" ]] || break
  FILES+=("$ARG")
  shift
done 

if
  [[ "${1-}" = "--" ]]
then
  shift
else
  echo "You must terminate the file list with -- to separate it from the command"
  (return, exit, whatever you prefer to stop the script/function)
fi

At this point, you have all the target files in an array, and "$@" contains only the command to execute. All that is left to do is :

for FILE in "${FILES[@]-}"
do
  cd "$FILE"
  "$@"
  cd "$CURRENT_DIR"
done

Please note that this solution has the advantage that if your user forgets the "--" separator, she will be notified (as opposed to a failure due to quoting).

Answer 2

In bash you may execute "set -o noglob" which will inhibit the shell to expand path names (globs). But this has to be set on the running shell before you execute the script, otherwise you should quote any meta character which you provide in the arguments.

Answer 3

find-while-read combination is one of the safest combination to parse file names. Do something like below

#!/bin/bash
myfunc(){
 cd "$2"
 eval "$1" # Execute the command parsed as an argument
}
cur_dir=$(pwd) # storing the current directory
find . -type d -print0 | while read -rd '' dname
do
 myfunc "pwd" "$dname"
 cd "$cur_dir" #Remember myfunc changes the current working dir, so you need this
done

Answer 4

In this case the problem is not the expansion of metacharacter, is just that your script has an undefined number of arguments of which the last one is the command to execute for all previous arguments.

#!/bin/bash
CMND=$(eval echo "\${$#}")        # get the command as last argument without arguments or
while [[ $# -gt 1 ]]; do          # execute loop for each argument except last one
     ( cd "$1" && eval "$CMND" )  # switch to each directory received and execute the command
     shift                    # throw away 1st arg and move to the next one in line
done

Usage: ./script.sh * pwd or ./script.sh * "ls -l"

To have the command followed by arguments (ex. ./script.sh * ls -l) the script has to be longer because each argument has to be tested if it's a directory until the command is identified (or backwards until a dir is identified).

Here is an alternative script that would accept the syntax: ./script.sh <dirs...> <command> <arguments...> For example: ./script.sh * ls -la

# Move all dirs from args to DIRS array
typeset -i COUNT=0
while [[ $# -gt 1 ]]; do
    [[ -d "$1" ]] && DIRS[COUNT++]="$1" && shift || break
done

# Validate that the command received is valid
which "$1" >/dev/null 2>&1 || { echo "invalid command: $1"; exit 1; }

# Execute the command + it's arguments for each dir from array
for D in "${DIRS[@]}"; do 
     ( cd "$D" && eval "$@" )
done

Answer 5

Here is how I would do it:

#!/bin/bash

# Read directory arguments into dirs array
for arg in "$@"; do
    if [[ -d $arg ]]; then
        dirs+=("$arg")
    else
        break
    fi
done

# Remove directories from arguments
shift ${#dirs[@]}

cur_dir=$PWD

# Loop through directories and execute command
for dir in "${dirs[@]}"; do
    cd "$dir"
    "$@"
    cd "$cur_dir"
done

This loops over the arguments as seen after expansion, and as long as they are directories, they are added to the dirs array. As soon as the first non-directory argument is encountered, we assume that now the command starts.

The directories are then removed from the arguments with shift, and we store the current directory in cur_dir.

The last loop visits each directory and executes the command consisting of the rest of the arguments.

This works for your

./do-for.sh repo-* git commit -a -m "Added new files."

example – but if repo-* expands to anything other than directories, the script breaks because it will try to execute the filename as part of the command.

It could be made more stable if, for example, the glob and the command were separated by an indicator such as --, but if you know that the glob will always be just directories, this should work.

Answer 6

Why not keep it simple and create a shell function that uses find but eases the burden for your users of typing out its commands, for example:

do_for() { find . -type d \( ! -name . \) -not -path '*/\.*' -name $1 -exec bash -c "cd '{}' && "${@:2}" " \;  }

So they can type something like do_for repo-* git commit -a -m "Added new files." Note, if you want to use the * by itself, you'll have to escape it:

do_for \* pwd 

Answer 7

Wildcards are evaluated by the shell before being passed to any program or script. There is nothing you can do about that.

But if you accept quoting the globbing expression then this script should to do the trick

#!/usr/bin/env bash

for dir in $1; do (
    cd "$dir"
    "${@:2}"
) done

I tried it out with two test directories and it seems to be working. Use it like this:

mkdir test_dir1 test_dir2
./do-for.sh "test_dir*" git init
./do-for.sh "test_dir*" touch test_file
./do-for.sh "test_dir*" git add .
./do-for.sh "test_dir*" git status
./do-for.sh "test_dir*" git commit -m "Added new files."

Answer 8

Nobody proposing a solution using find ? Why not try something like this:

find . -type d \( -wholename 'YOURPATTERN' \) -print0 | xargs -0 YOURCOMMAND

Look at man find for more options.

Answer 9

I will begin with the Windows batch file that you mentioned twice as working. The big difference is that on Windows, the shell doesn’t make any globbing, leaving it to the various commands (and each of them does it differently), while on Linux/Unix the globbing is usually done by the shell, and can be prevented by quoting or escaping. Both the Windows approach and the Linux approach have their merits, and they compare differently in different use cases.

For regular bash users, quoting

   ./do-for.sh repo-'*' git commit -a -m "Added new files."

or escaping

   ./do-for.sh repo-\* git commit -a -m "Added new files."

are the simplest solution, because they are what they consistently use on a daily basis. If your users need a different syntax, you have all the solutions proposed so far, that I will classify into four categories before proposing my own (note that in each example below do-for.sh stands for a different script adopting the respective solution, which can be found in one of the other answers.)

• Disable shell globbing. This is clumsy, because, even if you remember which shell option does it, you have to remember to reset it to default to have the shell working normally afterwards.

• Use a separator:

  ./do-for.sh repo-* -- git commit -a -m "Added new files."
  

This works, is similar to the solution adopted in similar situations with other shell commands, and fails only if your expansion of directory names includes a directory name exactly equal to the separator (an unlikely event, which wouldn’t happen in the above example, but in general might happen.)

• Have the command as the last argument, all the rest are directories:

  ./do-for.sh repo-* 'git commit -a -m "Added new files."'
  

This works, but again, it involves quoting, possibly even nested, and there is no point in preferring it to the more usual quoting of globbing characters.

• Try to be smart:

   ./do-for.sh repo-* git commit -a -m "Added new files."
  

and consider to be dealing with directories till you hit a name which is not a directory. This would work in many cases, but might fail in obscure ways (e.g. when you have a directory named like the command).

My solution doesn’t belong to any of the mentioned categories. In fact, what I propose is not to use * as a globbing character in the first argument of your script. (This is similar to the syntax used by the split command where you provide a non-globbed prefix argument for the files to be generated.) I have two versions (code below). With the first version, you would do the following:

        # repo- is a prefix: the command will be excuted in all
        # subdirectories whose name starts with it
        ./do-for.sh repo- git commit -a -m "Added new files."

        # The command will be excuted in all subdirectories
        # of the current one
        ./do-for.sh . git commit -a -m "Added new files."

        # If you want the command to be executed in exactly 
        # one subdirectory with no globbing at all, 
        # '/' can be used as a 'stop character'. But why 
        # use do-for.sh in this case?
        ./do-for.sh repo/ git commit -a -m "Added new files."

        # Use '.' to disable the stop character.
        # The command will be excuted in all subdirectories of the
        # given one (paths have to be always relative, though)
        ./do-for.sh repos/. git commit -a -m "Added new files."

The second version involves using a globbing character the shell knows nothing about, such as SQL’s % character

        # the command will be excuted in all subdirectories 
        # matching the SQL glob
        ./do-for.sh repo-% git commit -a -m "Added new files."
        ./do-for.sh user-%-repo git commit -a -m "Added new files."
        ./do-for.sh % git commit -a -m "Added new files."

The second version is more flexible, as it allows non-final globs, but is less standard for the bash world.

Here is the code:

#!/bin/bash
if [ "$#" -lt 2 ]; then
  echo "Usage: ${0##*/} PREFIX command..." >&2
  exit 1
fi

pathPrefix="$1"
shift

### For second version, comment out the following five lines
case "$pathPrefix" in
  (*/) pathPrefix="${pathPrefix%/}" ;;   # Stop character, remove it
  (*.) pathPrefix="${pathPrefix%.}*" ;;  # Replace final dot with glob
  (*) pathPrefix+=\* ;;                  # Add a final glob
esac
### For second version, uncomment the following line
# pathPrefix="${pathPrefix//%/*}"        # Add a final glob

tmp=${pathPrefix//[^\/]}   # Count how many levels down we have to go
maxDepth=$((1+${#tmp}))


# Please note that this won’t work if matched directory names
# contain newline characters (comment added for those bash freaks who 
# care about extreme cases)
declare -a directories=()
while read d; do
  directories+=("$d")
done < <(find . -maxdepth "$maxDepth" -path ./"$pathPrefix" -type d -print)

curDir="$(pwd)"
for d in "${directories[@]}"; do
  cd "$d";
  "$@"
  cd "$curDir"
done

As in Windows, you would still need to use quotes if the prefix contains spaces

        ./do-for.sh 'repository for project' git commit -a -m "Added new files."

(but if the prefix does not contain spaces, you can avoid quoting it and it will correctly deal with any space-containing directory names beginning with that prefix; with obvious changes, the same is true for %-patterns in the second version.)

Please note the other relevant differences between a Windows and a Linux environment, such as case sensitivity in pathnames, differences in which characters are considered special, and so on.