My Spring custom HealthIndicator not called when using actuator liveness and readiness endpoints

Question

Our Spring custom HealthIndicator bean's health() method is not being called when we use /actuator/health/readiness (e.g. testing it from postman: http://localhost:9743/actuator/health/readiness) or /actuator/health/liveness. When we use /actuator/health, our Spring custom HealthIndicator bean's health() method is called.

Note: I changed the server port for my springboot test app to 9743

Details:

Pom has the following

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.6.0</version>
</parent>
    
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

We have the following custom HealthIndicator. It's to check MarkLogic db health and ... I removed the MarkLogic part and mimic'ed its failure by throwing an exception in the health() method:

import org.springframework.boot.actuate.health.Health;
import org.springframework.boot.actuate.health.HealthIndicator;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class MarkLogicDBHealthIndicatorConfig {
    
    private final Logger logger = LoggerFactory.getLogger(MarkLogicDBHealthIndicatorConfig.class);
    
    @Bean
    public MarkLogicDBHealthIndicator marklogic() {
        
        logger.info("Entered MarkLogicDBHealthIndicatorConfig.marklogic().  Creating and returning new MarkLogicDBHealthIndicator()");
        
        return new MarkLogicDBHealthIndicator();
    }
}

class MarkLogicDBHealthIndicator implements HealthIndicator {
    
    private final Logger logger = LoggerFactory.getLogger(MarkLogicDBHealthIndicator.class);

    @Override
    public Health health() {
        
        logger.info("Entered MarkLogicDBHealthIndicator.health().");
        
        Health.Builder mlHealth;    
        
        try {
            
            // Do something that simulates marklogic being down (= just have a java method throw an exception)
            this.alwaysThrowException();

            mlHealth = Health.up();
            
            mlHealth = mlHealth.withDetail("db-host", "my-db-host");
            mlHealth = mlHealth.withDetail("db-port", "my-db-port");
            mlHealth = mlHealth.withDetail("db-check-time", 1234);
            
        } catch (Exception e) {
            
            logger.warn("{}-{}. DB HealthCheck failed!", e.getClass().getSimpleName(), e.getMessage(), e);
            
            mlHealth = Health.down(e);
            
            mlHealth = mlHealth.withDetail("db-host", "my-db-host");
            mlHealth = mlHealth.withDetail("db-port", "my-db-port");
            mlHealth = mlHealth.withDetail("db-check-time", 1234);
        }
        
        Health h = mlHealth.build();
        
        logger.info("Leaving MarkLogicDBHealthIndicator.health().  h = " + h.toString());
        
        return h;
    }
    
    private void alwaysThrowException() throws Exception {
        throw new MyException("error");
    }
}

I needed the following in application.yml for sending /actuator/health/readiness and /actuator/health/liveness (otherwise, an http 404 error results). Note these are not needed when sending /actuator/health:

management:
  endpoint:
    health:
      probes:
        enabled: true
      livenessState:
        enabled: true
      readinessState:
        enabled: true

When the application starts, I see the log showing the bean being created:

Entered MarkLogicDBHealthIndicatorConfig.marklogic().  Creating and returning new MarkLogicDBHealthIndicator()

Exposing 1 endpoint(s) beneath base path '/actuator'

When I send http://localhost:9743/actuator/health, I get the expected http status of 503 (in postman) and see my health() method being called from the log:

Entered MarkLogicDBHealthIndicator.health().

MyException-error. DB HealthCheck failed!
com.ibm.sa.exception.MyException: error

However, when I send http://localhost:9743/actuator/health/readiness or http://localhost:9743/actuator/health/liveness, my MarkLogicDBHealthIndicator health() method is NOT called.

Note: In our actual deployment, our applications are deployed to Kubernetes, and we specify the liveness and readiness endpoints in each application's deployment yaml (gen'ed using helm so ... easy to change). None of our applications do anything differently for readiness vs liveness so ... we could just switch to /actuator/health for both liveness and readiness & then I know it will work.

Answer 1

totally the same problem in spring boot 2.7.13 but with standard healthIndicators: actuator/health call shows DOWN but liveness and readiness are UP.

{
    "status": "DOWN",
    "components": {
        "db": {
            "status": "UP",
            "details": {
                "database": "MySQL",
                "validationQuery": "isValid()"
            }
        },
        "diskSpace": {
            "status": "UP",
            "details": {
                "total": 509856444416,
                "free": 106533199872,
                "threshold": 10485760,
                "exists": true
            }
        },
        "livenessState": {
            "status": "UP"
        },
        "ping": {
            "status": "UP"
        },
        "rabbit": {
            "status": "DOWN",
            "details": {
                "error": "org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused: no further information"
            }
        },
        "readinessState": {
            "status": "UP"
        }
    },
    "groups": [
        "liveness",
        "readiness"
    ]
}

It is very strange than liveness and readiness doesn't care about health state at all. Could it be bug of current version or it is wrong configuration?

I have the following configuration:

management.endpoint.health.show-details=always
management.endpoint.health.probes.enabled=true
management.health.livenessstate.enabled=true
management.health.readinessstate.enabled=true
spring.lifecycle.timeout-per-shutdown-phase=30s
server.shutdown=graceful
management.endpoint.shutdown.enabled=true
management.endpoints.web.exposure.include=shutdown, health
management.server.port=8080
management.server.ssl.enabled=false