Is it possible to create a long running process in NodeJs

Question

Is it possible to create a long running process in NodeJs to handle many background operations without interrupting the main thread; something like Celery in Python.

Hint, it's highly preferable to be able to manage that long-running process, in case of failure, or need to be restarted, away from the main process.

Answer 1

http://nodejs.org/api/child_process.html is the right API to create long-running processes, you will have complete control over the child processes (access to stdin/out/err, can send signals etc). This approach however requires that your node process is parent of those children.. If you want the child to outlive the parent, take a look at options.detached during child creation (and following child.unref()).

Please note, however, that Node.js is suited extremely well to avoid such architecture. Typically node.js do all the background stuff in the main thread. I've been writing apps with lots of traffic (like thousands requests per second), with DB, Redis and RabbitMQ access all from the main thread and without any child processes - and it was worked fine, as it should, thanks to Node's evented IO system.

I'm generally using child_process api only to launch separate executables (e.g. ffmpeg to transcode some video file), apart of such scenarios separate processes are probably not what you want.

There is also cluster api which allow single master to handle numerous worker processes, though I think it isn't what you look for, either.

Answer 2

You can create child process to handle your background operations. And then use messages to pass data between the new process and your main thread.

http://nodejs.org/api/child_process.html

Update

It looks like you need to use the server queues, sort of beanstalkd http://kr.github.io/beanstalkd/ + https://www.npmjs.com/package/fivebeans.