system: Resource temporarily unavailable, which one?

Question

I search for answer and so far haven't found a clear one.

I am doing testing which launches many threads calling "system()", like below.

for (int i = 0; i < 3000; ++i)
  pthread_create(&thread[i], NULL, thread_func, NULL);

for (int i = 0; i < 3000; ++i)
  pthread_join(thread[i], NULL);

...
void* thread_func(void* arg)
{
  if (system('test.sh') == -1)
  {
    perror("system");
    exit(1);
  }
  pthread_exit(NULL);
}

test.sh

#!/bin/bash

sleep 100

When I run the program, at certain point it will display.

system: Resource temporarily unavailable

Is there way to know which resource? I fix the max processes issue so I think it may be due to something else.

Answer 1

This error means that some system call called by the system library function returned EGAIN. Most likely is the fork call, which can fail with EAGAIN for a number of reasons:

   EAGAIN A system-imposed limit on the number of threads was encountered.  There are a  num‐
          ber of limits that may trigger this error:

          *  the  RLIMIT_NPROC  soft  resource limit (set via setrlimit(2)), which limits the
             number of processes and threads for a real user ID, was reached;

          *  the  kernel's  system-wide  limit  on  the  number  of  processes  and  threads,
             /proc/sys/kernel/threads-max, was reached (see proc(5));

          *  the maximum number of PIDs, /proc/sys/kernel/pid_max, was reached (see proc(5));
             or

          *  the PID limit (pids.max) imposed by the  cgroup  "process  number"  (PIDs)  con‐
             troller was reached.