I have just tried concurrent.futures module to play with concurrency and found that ThreadPoolExecutor is distributing the tasks according to the availability of the thread.
Is there any approach to distribute the tasks equally (like the round-robin technique) to ensure fairness?
code: https://colab.research.google.com/drive/1z4CWPX0wAqHgPXjyolPQ9rjob2R_Ma_M?usp=sharing
with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:
futures = {executor.submit(is_prime, number) for number in range(h_range + 1)}
The output of the code is given below. Here threads 0, 1, and 2 have done all the tasks. but expecting the tasks will be distributed to each of the threads equally and assign the next task according to availability to execute the program parallelly.
ThreadPoolExecutor-8_0
ThreadPoolExecutor-8_0
ThreadPoolExecutor-8_2
ThreadPoolExecutor-8_2
ThreadPoolExecutor-8_2
ThreadPoolExecutor-8_1
ThreadPoolExecutor-8_1
ThreadPoolExecutor-8_1
ThreadPoolExecutor-8_1
ThreadPoolExecutor-8_0
ThreadPoolExecutor-8_1
Background
In the shared code thread_pool_with_chunked method has used all the threads and returned the output in 25 sec. But the first chunk has completed the task within 10 sec and it was idle the rest of the execution time. So, to make the code more efficient, the tasks should be distributed in threads equally so that no thread can be idle in the execution time. That is the goal of this question.