File sync with node.js: unision== tcp==node-----[http]-----node==tcp==unison

Question

If you have node running and I have node running, has anyone seen a solution that will allow us to keep a set of files in sync.

Synchronisation is complicated, so I thought that could be left to a tool like unison (which is like rsync) and then all node would have to do is connect up a TCP pipe between authenticated users.

filesystem1---unision==tcp==node.js------[http]----node.js==tcp====unison---filesystem2

It's probably about 12 lines of JavaScript, but at this point it is beyond me, or any of the examples I could find, so far.

I have looked at a whole bunch of other file synchronisation options (like Git, veracity, fossil, including a week of trying to install a Simias iFolder server on Linux, fail ~ which looked promising because it included a filewatching client for each major OS) but now I'm thinking that something much, much simpler would probably be the go.

If anyone has seen a Node.js project that does such, or is at a level where connecting two TCP pipes is not too hard, then I would appreciate hearing from you

Answer 1

The basic approach I would take to this problem is to use stream2.
My basic approach would be something like this.

getting data from first tcp

var gulp = require('gulp')
, thr = require('through2')
;


gulp.watch('rootDirectory/**/*', function(evt) {


  if ('deleted' === evt.type) {
    // do stuff if file is deleted
    return
  }

  // file got changed, moved or added.
  gulp.src(evt.path).pipe(thr(chunk, enc, next){
    // chunk is a vinyl filesytem object, 
    // so better fix chunk before sending to tcp server
    next(null, fixedChunk)

  }).pipe(toYourTcpServer)

})

then at your node

var net = require('net')
, http = require('http')
;

var clientTcp = net.connect({port: 'toYourTcpServerPort'})

var server = http.createServer(function(req, res){
  clientTcp.pipe(res)
}).listen('somePort')

then on the other end, you do the reverse
at node

var clientTcp2 = net.connect({port: 'toYourTcpServerPort2'})

var server2 = http.createServer(function(req, res){
  // fix response or something else.
  req.pipe(clientTcp2)
}).listen('somePort')

coming from the second tcp

gutil = require('gulp-util')


clientTcp2.pipe(thr(function(chunk, enc, next){
  // You must create a vinyl filesystem before
  // sending down stream. Use new gutil.File()
  next(null, fixedChunk)

})).pipe(gulp.dest('finalDirectoy'))

through2 creates transforms streams and gulp help you create tasks with stream. Vinyl files are just an abstraction used by gulp.

I hope that gives you an idea of how to use streams to solve your problem good luck