Pipe rsync output to file

Question

Is it possible to pipe the output (raw data, no logging) to a file of rsync, and maybe save this to a bsdiff? I find rsnapshot to be highly irritating with the lack of configuration and stupid syntax (indenting, wtf?) and rdiff to be a little unstable.

I have a working solution where i rsync all the data over, make a tar.gz-archive of it and bsdiff to generate a patch between two tar.gz-archives. But this is pretty CPU intensive on huge loads and very disk intensive, as you have to make the entire archive every time.

To sum it up: - Make the initial rsync - bsdiff it against the previous files - Archive the diff in a way to make it easy to recover

When i wrote this i just got an idea with lvm-snapshot, any takers on how I should go forth with that?

Answer 1

rsync(1) can use a different shell easily when contacting remote systems; -e ssh is the default, but you could write a wrapper program that starts ssh and saves a copy of all the content that you forward along to ssh.

Some pseudo-code for what I'm thinking of:

in[2] = pipe
out[2] = pipe
err[2] = pipe
fork
if child, fork
grandparent /* ssh */
    close 0, 1, 2
    dup2 in[0], 0
    dup2 out[1], 1
    dup2 err[1], 2
    close in, out, err
    exec(ssh, hostname, rsync) -- copy from rsync itself
parent /* stdin -> pipe */
    close in[0], out, err, 1, 2
    open (log, "/path/to/log", "w")
    loop forever:
        read(0, buf)
        write(in[1], buf)
        write(log, buf)
child /* pipe -> stdout, stderr */
    close in, out[1], err[1], 0
    loop forever:
        read(out[0], bufout)
        write(1, bufout)
        read(err[0], buferr)
        write(2, buferr)

Please double-check the pipe(2), dup2(2), and close(2) calls before implementing this; I believe I hooked up the descriptors properly and closed the unused descriptors in each process, but this is a little subtle.

I don't know how easy it will be for you to apply the resulting file to your remote endpoint's tarball, but I hope that's not the part you're stuck on.