gRPC C++ client blocking when I set a deadline time

Question

In my grpc client, it sends a keepalive packet every second.The code like this:

bool FcrConfigClient::SendHealthCheckRpc(const grpc::string& service_name)
{
    CtlHealthCheckRequest request;
    unsigned int client_connection_timeout = HEALTH_CHECK_TIMER_OUT;
    CtlHealthCheckResponse response;
    ClientContext context;
    CtlFcrStatus *fcr_status;
    std::chrono::system_clock::time_point deadline = std::chrono::system_clock::now() + std::chrono::seconds(client_connection_timeout);
    context.set_deadline(deadline);

    request.set_service(service_name);
    request.set_requestoffline(requestOffline);
    fcr_status = request.mutable_fcr_status();
    fcr_status->set_prober_status(global_grpc_shm_config->proberResult);
    fcr_status->set_offline_flag(global_grpc_shm_config->offline);

    Status s = stub_->check(&context, request, &response);
    ...... 

When I config iptables in Linux to drop all the grpc packets, the grpc channel was blocked. The stack is:

(gdb) bt
#0  0x00007ffff5e8f873 in epoll_wait () from /lib64/libc.so.6
#1  0x00007ffff69164c8 in pollable_epoll (deadline=<optimized out>, p=0x7fffe4001890)
    at src/core/lib/iomgr/ev_epollex_linux.cc:823
#2  pollset_work (pollset=0x808570, worker_hdl=<optimized out>, deadline=<optimized out>)
    at src/core/lib/iomgr/ev_epollex_linux.cc:1010
#3  0x00007ffff693a651 in cq_pluck (cq=0x808470, tag=0x7fffffffd990, deadline=..., reserved=<optimized out>)
    at src/core/lib/surface/completion_queue.cc:1172
#4  0x00007ffff693a9db in grpc_completion_queue_pluck (cq=<optimized out>, tag=<optimized out>, deadline=..., 
    reserved=<optimized out>) at src/core/lib/surface/completion_queue.cc:1199
#5  0x00007ffff750a9d8 in grpc::CoreCodegen::grpc_completion_queue_pluck (this=<optimized out>, 
    cq=<optimized out>, tag=<optimized out>, deadline=..., reserved=<optimized out>)
    at src/cpp/common/core_codegen.cc:70
#6  0x0000000000477662 in grpc::CompletionQueue::Pluck (this=0x7fffffffd950, tag=0x7fffffffd990)
    at /usr/local/include/grpcpp/impl/codegen/completion_queue.h:297
#7  0x00000000004f83c2 in grpc::internal::BlockingUnaryCallImpl<fcrctlservice::CtlHealthCheckRequest, fcrctlservice::CtlHealthCheckResponse>::BlockingUnaryCallImpl (this=0x7fffffffdb00, channel=0x7fffe00103c0, method=..., 
    context=0x7fffffffdbd0, request=..., result=0x7fffffffdd70)
    at /usr/local/include/grpcpp/impl/codegen/client_unary_call.h:72
#8  0x00000000004f5bd9 in grpc::internal::BlockingUnaryCall<fcrctlservice::CtlHealthCheckRequest, fcrctlservice::CtlHealthCheckResponse> (channel=0x7fffe00103c0, method=..., context=0x7fffffffdbd0, request=..., 
    result=0x7fffffffdd70) at /usr/local/include/grpcpp/impl/codegen/client_unary_call.h:43
#9  0x00000000004f3eca in fcrctlservice::FcrCtlService::Stub::check (this=0x7fffe0006450, context=0x7fffffffdbd0, 
    request=..., response=0x7fffffffdd70) at fcr_config_client.grpc.pb.cc:73
#10 0x00000000004bdd17 in FcrConfigClient::SendHealthCheckRpc (this=0x7fffe0006290, service_name="16.2.6.138")
    at fcr_config_grpc.cc:380
#11 0x00000000004cc315 in fcr_grpc_health_check_timer (t=0x7fffffffdff0) at fcr_config_grpc.cc:3045
#12 0x00007ffff7274295 in thread_call (thread=0x7fffffffdff0) at lib/thread.c:1542
#13 0x00000000004cfb0b in RunGrpcServer () at fcr_config_grpc.cc:4205
#14 0x00000000004d037b in main (argc=1, argv=0x7fffffffe3d0) at fcr_config_grpc.cc:4378
(gdb)

I search the grpc::CompletionQueue::Pluck function in source code, it defines like this:

  bool Pluck(internal::CompletionQueueTag* tag) {
    auto deadline =
        g_core_codegen_interface->gpr_inf_future(GPR_CLOCK_REALTIME);
    auto ev = g_core_codegen_interface->grpc_completion_queue_pluck(
        cq_, tag, deadline, nullptr);
    bool ok = ev.success != 0;
    void* ignored = tag;
    GPR_CODEGEN_ASSERT(tag->FinalizeResult(&ignored, &ok));
    GPR_CODEGEN_ASSERT(ignored == tag);
    // Ignore mutations by FinalizeResult: Pluck returns the C API status
    return ev.success != 0;
  }

The deadline which I set in my code doesn't be used in Pluck function. So my question is why the grpc client blocked, did deadline time not take effect?

Answer 1

You might be running into the same issue as https://github.com/grpc/grpc/issues/15889.

Please make sure the KEEPALIVE_TIMEOUT setting (GRPC_ARG_KEEPALIVE_TIMEOUT_MS) is configured properly.