Getting continue behavior (not redownloading files already present) using lftp.

Question

So, I have a script which downloads stuff from a seedbox. It works great for new files which are in the remote server and then mirrored on my local server. The problem is that when I want, for example, to remove unnecessary files, running the script again re-downloads the same file(s) again. I tried going into the man pages of mirror but it wasn't helpful. Here is the script which mirrors the files:

#!/bin/bash

login=XXXX
pass=XXXXXX
host=XXXXX
remote_dir=/files/
local_dir=/home/XXX/XXX

trap "rm -f /tmp/seedroots.lock" SIGINT SIGTERM

if [ -e /tmp/seedroots.lock ]; then
   echo "Synctorrent is running already."
   exit 1
else
   touch /tmp/seedroots.lock
   lftp -p 21 -u $login,$pass $host << EOF
   set ftp:ssl-allow no
   set mirror:use-pget-n 5
   mirror -c -P5 --log=synctorrents.log $remote_dir $local_dir

   EOF

   rm -f /tmp/seedroots.lock
   exit 0
fi

Is there an option for mirror which I am missing that doesn't re-download the locally deleted file(s) again?

Answer 1

You should give a try to my version of your script (not tested) :

#!/bin/bash

login=XXXX
pass=XXXXXX
host=XXXXX
remote_dir=/files/
local_dir=/home/XXX/XXX
files=$local_dir/*

trap "rmdir /tmp/seedroots.lock" 0 1 2 3 15

if [[ -d /tmp/seedroots.lock ]]; then
    echo "Synctorrent is running already."
    exit 1
else
    mkdir /tmp/seedroots.lock

    lftp -p 21 -u $login,$pass $host << EOF
    set ftp:ssl-allow no
    set mirror:use-pget-n 5
    mget $files

    EOF

fi

What it does :

• I build a local list of files, and, subsequently, mget all these files on the ftp server with the variable $files.
• I replaced the lock file with a dir : search web about atomicity.
  Files are not atomic whereas directories are.
• The trap runs on normal exit and other signals
• If you are using bash, [[ ]] tests are more powerfull.
• Indentation is not just an option ;)

Answer 2

The mirror command in lftp has a --continue flag which will result in the behavior you want.

Answer 3

If you are just leeching files (not seeding), you can use lftp mirror with --Remove-source-files option to remove files at source after transfer (so no duplicate, re-downloads).