Shutting down nodejs microservice gracefully when using Docker Swarm

Question

I am have multiple micro services written in Nodejs Koa running in Docker Swarm.

Since container orchestration tools like Kubernetes or Swarm can scale up and down services instantly, I have a question on gracefully shutting down Nodejs service to prevent unfinished running process.

Below is the flow I can think of:

1. sending a SIGNINT signal to each worker process, Does docker swarm send SIGNINT to worker when scaling down service?

2. the worker are responsible to catch the signal, cleanup or free any used resource and finish the its process, How can I stop new api request, wait for any running process to finish before shutting down?

   Some code below from reference:

   process.on('SIGINT', () => {
     const cleanUp = () => {
       // How can I clean resources like DB connections using Sequelizer
     }
   
     server.close(() => {
   
       cleanUp()
       process.exit()
     })
   
     // Force close server after 5secs, `Should I do this?`
     setTimeout((e) => {
       cleanUp()
       process.exit(1)
     }, 5000)
   })
   

Answer 1

I created a library (https://github.com/sebhildebrandt/http-graceful-shutdown) that can handle graceful shutdowns as you described. Works well with Express and Koa.

This package also allows you to create function (should return a promise) to additionally clean up things like DB stuff, ... here some example code how to use it:

const gracefulShutdown = require('http-graceful-shutdown');
...
server = app.listen(...);
...

// your personal cleanup function - this one takes one second to complete
function cleanup() {
  return new Promise((resolve) => {
    console.log('... in cleanup')
    setTimeout(function() {
        console.log('... cleanup finished');
        resolve();
    }, 1000)       
  });
}

// this enables the graceful shutdown with advanced options
gracefulShutdown(server,
    {
        signals: 'SIGINT SIGTERM',
        timeout: 30000,
        development: false,
        onShutdown: cleanup,
        finally: function() {
            console.log('Server gracefulls shutted down.....')
        }
    }
);

I personally would increase the final timeout from 5 secs to higher value (10-30 secs). Hope that helps.