I need to transfer data between machine A (linux filesystem) and machine B (Ceph/S3 object filesystem) from machine C (the only one I can control). On machine C, I installed aws cli for my transfers.
I've worked out how to transfer from A to B: scp user@A:/path/to/file >(aws s3 cp - s3://s3uri/)
But I'm stumped for the transfer from B to A. I've tried lots of things without success. The command aws s3 cp s3://s3uri/filename.txt >(scp - user@A:/path/to/file) gives me this error:
-: No such file or directory
download failed: s3://s3uri/filename.txt to ../../dev/fd/63 [Errno 32] Broken pipe
it would seem that notation with a hyphen is not permitted in this writing style
The command scp <(aws s3 cp s3://s3uri/filename.txt -) user@A:/path/to/file) gives me this error:
/dev/fd/63: not a regular file
download failed: s3://s3uri/filename.txt to - [Errno 32] Broken pipe
Has anyone done this before?
Thanks for your help Thierry
You can workaround by decomposing scp into a succession of ssh and cat :
aws --quiet s3 cp ... >(cat) | ssh [serveur] 'cat > file.txt'When I test, the intermediate cat seems necessary for me else ssh reads the file content as the connection password which fails. --quiet is necessary for me to avoid aws to write "N/N bytes transferred [...]" at the end of the new file.
If you can use sshpass to automatically pass the password, then you can remove these cat and --quiet :
aws s3 cp ... >(sshpass -f file_with_password ssh [serveur] 'cat > file.txt')