Is gevent pool wait_available thread safe

Question

So I have a function lets say test and gevent pool of size 10.

pool.Pool(size=10)
def test():
    pool.wait_available(timeout=0.5)
    pool.spawn(something)

If the function test is called by different threads, will it cause any issue. I mean wait_availableshould be thread safe but the pool.spawn will it have thread safety/race condition issue. I mean lets say there are already 9 greenlets running and there a couple of requests calling the test function. Both of them will read pool.wait_available which should not block But the pool.spawn right after that will make one of them block.

I just want to make sure that the pool.spawn doesn't block for more than the specified timeout period. How can I accomplish this?

Answer 1

wait_available isn't necessary if you're using spawn from pool because spawn will ask for a lock in the Pools internal semaphore that is used to track running greenlets. An exception to this is only when you use apply_async. I will explain both scenarios here:

pool.Pool(size=10)
def test():
    for i in xrange(20):
        log('Processing {}'.format(i))
        pool.spawn(something)
    pool.join()
    log('Done')

The output of this shows it will spawn greenlets in groups of 10 since the pool contains space for 10:

1531531331: Processing 0
1531531331: Processing 1
1531531331: Processing 2
1531531331: Processing 3
1531531331: Processing 4
1531531331: Processing 5
1531531331: Processing 6
1531531331: Processing 7
1531531331: Processing 8
1531531331: Processing 9
1531531340: Processing 10
1531531340: Processing 11
1531531340: Processing 12
1531531340: Processing 13
1531531340: Processing 14
1531531340: Processing 15
1531531340: Processing 16
1531531340: Processing 17
1531531340: Processing 18
1531531340: Processing 19
1531531349: Done

Conversely, if you use apply_Async instead of spawn, it will force all the calls to run at the same time. There will be a race condition here for all greeenlets to start execution immediately.

1531531357: Processing 0
1531531357: Processing 1
1531531357: Processing 2
1531531357: Processing 3
1531531357: Processing 4
1531531357: Processing 5
1531531357: Processing 6
1531531357: Processing 7
1531531357: Processing 8
1531531357: Processing 9
1531531357: Processing 10
1531531357: Processing 11
1531531357: Processing 12
1531531357: Processing 13
1531531357: Processing 14
1531531357: Processing 15
1531531357: Processing 16
1531531357: Processing 17
1531531357: Processing 18
1531531357: Processing 19
1531531367: Done

If you use wait_available() at the beginning, you go back to similar behaviour to spawn. So, if you are using spawn, you dont need wait_available() as they do same check(checking the semaphore to see if there's any room in the pool).

Hope it helps! Gevent is amazing! happy coding!