Spring Cloud Gateway 500 when an instance is down

Question

I have a Spring Cloud Gateway (eureka client) app that uses Spring Cloud Load Balancer (Spring Cloud version: Hoxton.SR6) and I have an instance of a spring boot app (spring boot 2.3 with enabled graceful shutdown, (eureka client).

When I shutdown a spring boot service and perform a request through the gateway then the gateway throws 500 error (connection refused), instead of 503. 503 appears after a 1-2 minutes.

Can anyone clarify if it is an expected behavior?

It seems that the problem comes from eureka-client (1.9.21 version in my case) AtomicReference<Applications> localRegionApps isn't frequently updated

Thanks!

UPDATE: I decided to check deeper this 500 error. The result is that my system (ubuntu) gives this error if the port is not used:

curl -v localhost:9722
 Rebuilt URL to: localhost:9722/
   Trying 127.0.0.1...
 TCP_NODELAY set
 connect to 127.0.0.1 port 9722 failed: Connection refused
 Failed to connect to localhost port 9722: Connection refused
 Closing connection 0

So I put in my application.yml:

spring:
  cloud:
    gateway:
      routes:
        - id: my_route
          uri: http://localhost:9722/

Then when my request is routed to my_route and none of apps uses 9722 then I get an error:

io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: localhost/127.0.0.1:9722
    Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
    |_ checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
    |_ checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
    |_ checkpoint ⇢ HTTP GET "/internal/mail/internal/health-check" [ExceptionHandlingWebHandler]
Stack trace:
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
    at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124)
    at io.netty.channel.unix.Socket.finishConnect(Socket.java:251)
    at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:672)
    at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:649)
    at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:529)
    at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:465)
    at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:834)

It seems to be an unexpected exception, since it isn't possible to handle it using a circuit breaker or any gateway filter.

Is it possible to handle this error correctly? I would like to return 503 in this case

Answer 1

hava a try to define your custom ErrorWebExceptionHandler. see: org.springframework.boot.web.reactive.error.ErrorWebExceptionHandler org.springframework.boot.autoconfigure.web.reactive.error.DefaultErrorWebExceptionHandler

Answer 2

One of the easiest ways to map particular exception to particular HTTP status code is providing a custom bean of type org.springframework.boot.web.reactive.error.ErrorAttributes. Here is an example:

@Bean
public ErrorAttributes errorAttributes() {
    return new CustomErrorAttributes(httpStatusExceptionTypeMapper);
}

public class CustomErrorAttributes extends DefaultErrorAttributes {
    @Override
    public Map<String, Object> getErrorAttributes(ServerRequest request, ErrorAttributeOptions options) {
        Map<String, Object> attributes = super.getErrorAttributes(request, options);
        Throwable error = getError(request);
        MergedAnnotation<ResponseStatus> responseStatusAnnotation = MergedAnnotations
            .from(error.getClass(), MergedAnnotations.SearchStrategy.TYPE_HIERARCHY).get(ResponseStatus.class);
        HttpStatus errorStatus = determineHttpStatus(error, responseStatusAnnotation);
        attributes.put("status", errorStatus.value());
        return attributes;
    }

    private HttpStatus determineHttpStatus(Throwable error, MergedAnnotation<ResponseStatus> responseStatusAnnotation) {
        if (error instanceof ResponseStatusException) {
            return ((ResponseStatusException) error).getStatus();
        }
        return responseStatusAnnotation.getValue("code", HttpStatus.class).orElseGet(() -> {
           if (error instanceof java.net.ConnectException) {
               return HttpStatus.SERVICE_UNAVAILABLE;
           }
           return HttpStatus.INTERNAL_SERVER_ERROR;
        }
    }
}

Answer 3

You should use Cloud Circuit Breaker.

For that:

1. Declare corresponding starter in your pom:

   <dependency>
       <groupId>org.springframework.cloud</groupId>
       <artifactId>spring-cloud-starter-circuitbreaker-reactor-resilience4j</artifactId>
    </dependency>
   

2. Declare circuit breaker in application.yaml

   spring:
     cloud:
       gateway:
         routes:
           - id: my_route
             uri: http://localhost:9722/
             filters:
               - name: CircuitBreaker
                 args:
                   name: myCircuitBreaker
                   fallbackUri: forward:/inCaseOfFailureUseThis
   

3. Declare the endpoint which will be called in the case of failure (a connection error, for example)

   @RequestMapping("/inCaseOfFailureUseThis")
       public Mono<ResponseEntity<String>> inCaseOfFailureUseThis() {
           return Mono.just(ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE).body("body for service failure case"));
   }