socket communication time cost too long

Question

I'm running a piece of program on a Linux development board, and I found that when the CPU load is low, it works alright, but if the CPU load gets to a high peak, it will take a much longer time.

Here is how it looks like: There are 2 programs running on board. Comsumer application has multiple threads, and they will call func1 to request some info from the producer process. The producer is a daemon process that will feed info back to the Consumer process.

The sample code looks like this :

Comsumer:

static int send_to_service(msg_t *cmd)
{
    int cnt = send(fd, cmd, sizeof(msg_t), MSG_WAITALL);
    if (cnt != sizeof(msg_t)) {
        log(L_ERROR, "send failed");
        return -1;
    }
    return 0;
}


static int func1(int aaa)
{
    struct_t msg = {...};
    msg.a = ...;
    msg.b = ...;    
    gettimeofday(time1, NULL);
    send_to_service(&msg);

    ...
}

Producer:

while(1) {

    gettimeofday(time2, NULL);
    int ret = select(maxfd+1, &fd_list, NULL, NULL, NULL);  
    gettimeofday(time3, NULL);
    for (int fd = 0; fd <= maxfd; fd ++) {
        if (!FD_ISSET(fd, &fd_list))                                            
           continue;
        ...
    }
}

The time difference between time1 and time3 can be 30ms+ more during CPU high load time. And it does not happen all the time, only once in a while.

I tried the single process way earlier, that way the Consumer calls the driver and get the info directly. That worked well. Now I have to add another process to the system to get the same info, so I have to use a daemon process to feed to both process. The performance is not as good as the single process method. even if there is only one consumer.

The system I use is Linux version 4.14.74, I'm not sure about the socket type and network, both consumer processes are within the same system waiting to get image info. I just used "send, recv and select" system provided.