Graceful shutdown in Vetx Application

Question

Scaling down the Kubernetes pod is not closing the Vertx application(developed in java) gracefully. It basically kills the container.

Overridden the beforeStoppingVertx and afterStoppingVertx methods of io.vertx.core.Launcher class and printed some logs to verify Vertx is shutting down gracefully. When the pods are scaled down, the methods beforeStoppingVertx and afterStoppingVertx are never being invoked.

  @Override
  public void beforeStoppingVertx(Vertx vertx) {
    LOG.info("Vertx Closing Gracefully");
    vertx.close();
  }

  @Override
  public void afterStoppingVertx() {
    LOG.info("Vertx Closed Gracefully");
  }

Tried adding grace-period of 60 seconds to the deployments yml which is the common template, still it is of no use ( beforeStoppingVertx and afterStoppingVertx methods not invoked even after 60 secs)

My suspicion is that vert.x is never getting told to shut down. Or it's not getting the message. We do have options to define Pre Stop hooks in Kubernetes, https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination

The question is what is the actual script, that we need to mention as part of the Pre-stop hook, that would instruct the Vert.x to shutdown gracefully ?

Answer 1

IMHO, rather than Kubernetes Vertx doesn't even trigger vertx.close() since you're using a custom Launcher class.

Try adding the following shutdown hook at the end of your main method:

public static void main(String[] args) {

    final Vertx vertx = Vertx.vertx();
    
    // deploy your verticles...

    Runtime.getRuntime().addShutdownHook(() -> {
        try {
          this.vertx.close()
            .toCompletionStage()
            .toCompletableFuture()
            .get(5, TimeUnit.SECONDS);

          log.info("<main> graceful shutdown success");
        } catch (Exception e) {
          log.error("<main> graceful shutdown error", e);
        }
    });
  }