Remote directory compare and merge without ssh

Question

I have 2 remote servers/machine say s1 and s2 (linux based machines) Both the server has 1 directory which is very huge. (i mean initially same data in both machines)

s1 is always stable upto date, changes are added by authorized user.
s2 people will make changes to the data here and there.

now requirement is to make content of s2 to inSync with s1.

Condition:

1. No replacement of s2 content with s1 because data is very huge
2. No other software allowed to install in machines
3. Only scp, sftp supported, no ssh or any other sort of access is given because it is production machine.

If anybody come across this sort of requirement Please suggest me any tool, any way to do this task.

Answer 1

If, you say, you have scp, then you must also have ssh. scp requires ssh to work. So, I'll start by challenging your assumption that you can't use rsync over ssh. If you have scp working, then there's no reason why rsync over ssh should not work.

rsync over ssh is the correct answer here. This is the most efficient mechanism for synchronizing content between two different servers. But, I suppose that it's possible that someone who thinks he knows what he's doing, but he really doesn't, hacked up a server to allow only the scp service, and block ssh sessions. Probably under a mistaken notion that this improves security somehow. It really doesn't, but that's a different topic. So, what now...

Well, you say you do have sftp access available. In that case, the next best answer would be a custom sftp client. Learn perl, and use the Net::SFTP module to write a custom perl script, for your specific requirement, to use SFTP to compare the contents of the two servers, and synchronize their contents.

Net::SFTP exposes the underlying SFTP protocol in a way that allows one to write custom applications that uses it. You'll use the SFTP protocol to examine the contents of each server, figure out what's different, then copy what needs to be copied, in order to update their contents.

Using Net::SFTP won't be as efficient as using rsync+ssh. With Net::SFTP, you'll know which files exist on the server, and the size of each file in bytes. However, if both servers appear to have a file with the same name, and the same byte count, you don't really know whether they are, in fact, identical, without downloading each file, and manually comparing them. You'll have to do that, of course. This is the key advantage of rsync+ssh that you do not have an sftp equivalent of. The rsync server works together with the rsync client, and they're able to verify that the file contents are identical, using checksums, without actually transferring the file from one side to another. No way to avoid doing that with sftp in this case, but this is going to be the best you'll be able to do.

Answer 2

If you decide to go the Perl route, don't use Net::SFTP which is an old an unmaintained module. Instead go for Net::SFTP::Foreign that BTW, implements recursive downloads allowing you to select which files to get on the fly, so you can easily do an update.

Another alternative is to use the development version of my other module Net::SSH::Any that has a built-in scp client that is able to download only the files that are newer on the remote side:

my $ssh = Net::SSH::Any->new(...);
$ssh->scp_get( { recursive => 1,
                 update => 1 },
               $remote_dir, $local_dir );

Other scripting languages like Python or Ruby also have SFTP and SCP libraries.