grpc io exception in k8s prod env

Question

i used grpc in java program and it work very well in the test env.But when we deployed in the production env it sometimes occurs exception with stack trace:

io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
        at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.34.1.jar!/:1.34.1]
        at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.34.1.jar!/:1.34.1]
        at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:123) ~[grpc-stub-1.34.1.jar!/:1.34.1]
        at cn.vpclub.sd.framework.grpc.client.proxy.ManageChannelProxy.invoke(ManageChannelProxy.java:107) ~[vpsd-grpc-1.3.0-cmpm-SNAPSHOT.jar!/:1.3.0-cmpm-SNAPSHOT]
        at org.springframework.cglib.proxy.Proxy$ProxyImpl$$EnhancerByCGLIB$$83232b9b.orderSyncFromBoss(<generated>) [spring-core-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
        at cn.vpclub.cmpm.capacity.central.bff.api.ordersync.service.OrderSyncFromBossService.orderSyncFromBoss(OrderSyncFromBossService.java:121) [classes!/:1.0-SNAPSHOT]
        at cn.vpclub.cmpm.capacity.central.bff.api.ordersync.controller.OrderSyncFromController.orderSync(OrderSyncFromController.java:45) [classes!/:1.0-SNAPSHOT]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_202]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_202]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_202]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_202]
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) [spring-web-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) [spring-web-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105) [spring-webmvc-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878) [spring-webmvc-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792) [spring-webmvc-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) [spring-webmvc-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040) [spring-webmvc-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943) [spring-webmvc-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) [spring-webmvc-5.2.15.RELEASE.jar!/:5.2.15.RELEASE]
[enter image description here](https://i.stack.imgur.com/IAIQX.jpg)

i build the ManagedChannel as this:

this.channelNetty = NettyChannelBuilder.forAddress(channelProperty.getHost(), channelProperty.getPort())
                .usePlaintext()
                .proxyDetector(proxyDetector)
                .maxInboundMessageSize(channelProperty.getMaxInboundMessageSize())
                .intercept(new GrpcClientInterceptor())
                .keepAliveTime(5, TimeUnit.MINUTES)
                .keepAliveTimeout(channelProperty.getDeadline(), TimeUnit.SECONDS)
                .build();

and the tcp keepalive is like this[tcp keepalive]

(https://i.stack.imgur.com/0GzKS.png)

we deploy in k8s and the grpc port is 50051,what can i do to solve this problem