S3FS - Recursive CHOWN/CHMOD takes a LONG time

Question

Any recursive chown or chmod command on an s3fs mount takes a long time when you have a few directories (about 70) each with quite a few files.

Either of these commands are likely to take almost 24 hours. I have to do this or the Apache process cannot access these files/directories. The command on a normal mount takes about 20 seconds.

Mounting with:

/storage -o noatime -o allow_other -o use_cache=/s3fscache -o default_acl=public-read-write

In /etc/fuse.conf:

user_allow_other

Using latest version: 1.78

Any thoughts on how to do this faster?

Answer 1

After a while, I found it just better to parallel the processes in order to speed it up. Example:

find /s3fsmount/path/to/somewhere -print | xargs --max-args=1 --max-procs=100 chmod 777

It is still slow, but nowhere near as slow as it was.

Answer 2

Using aws cli may help.

what I do:

1. use aws cli to get the full file list of the target directory.
2. write a script to parallel execute chmod 777 to each file (with > /dev/null 2>&1 &)

then I found that the chmod jobs finished immediately, from ps -ef.

my PHP code:

<?php

$s3_dir = 'path/to/target/';
$s3fs_dir = '/mnt/s3-drive/' .$s3_dir;
echo 'Fetching file list...' . "\n\n";
sleep(1.5);

$cmd = 'aws s3 ls --recursive s3://<bucket_name>/' . $s3_dir;
exec($cmd, $output, $return);

$num = 0;
if ( is_array($output) ) {
    foreach($output as $file_str) {
        if ( $num>100 ) {
            sleep(4);
            $num=0;
        }

        $n = sscanf( $file_str, "%s\t%s\t%s\t". $s3_dir ."%s", $none1, $none2, $none3, $file );
        $cmd = 'chmod 777 ' . $s3fs_dir . $file . ' > /dev/null 2>&1 &';
        echo $cmd ."\n";
        exec( $cmd );
        $num+=1;
    }
}

?>

Answer 3

For Change user

find /s3fsmount/path/to/somewher -print | xargs --max-args=1 --max-procs=100 sudo chown -R  user:user

It work me..

Answer 4

Here is an enhancement to @jafo (https://stackoverflow.com/a/31271219/4867575) answer in case only a few files need to be chowned : As changing owner or group of a file through S3FS requires re-uploading the entire file this is why it takes extremely much time (see : https://stackoverflow.com/a/66080786/4867575).

When only a few files need to be changed you can select only the files that need be changed (inspired by https://stackoverflow.com/a/65218413/4867575) :

my_directory=<my_directory>
my_user=<my_user>
my_group=<my_group>

find $my_directory \( ! -user $my_user -o ! -group $my_group \) -print0 | xargs -0 --max-args=1 --max-procs=10 chown -v $my_user:$my_group