BrokenProcessPool error - Help me figure out why

46 Views Asked by At

Re-post to fix a couple of typographical errors.

I'm trying to work through some training videos for asynchronous processing. My environment is a local Jupyter Notebook with the necessary modules installed.

Here's the code

from concurrent.futures import ProcessPoolExecutor
from concurrent.futures import as_completed
import urllib.request
import time

url_list = ['http://www.looneycorn.com/', 'http://reuters.com/', 'http://wwf.panda.org', 'https://en.unesco.org/']

def url_loader(url, time):
    with urllib.request.urlopen(url, timeout = time) as conn:
        return conn.read

def main_processpool():
    start = time.time()
    with ProcessPoolExecutor(max_workers = 1) as executor:
        future_to_page = {executor.submit(url_loader, url, 60): url for url in url_list}
        for future in as_completed(future_to_page):
            url = future_to_page[future]
            result = future.result() #this is the line that causes the error
            print('The page %r is %d bytes' % (url, len(result)))
        print('Total time taken:', time.time() - start)

main_processpool()

When I run it then the error I get is

A process in the process pool was terminated abruptly while the future was runnning or pending

I've tried several things.

  1. Trying only a single URL doesn't make it work
  2. print(future.done()) right before result = future.result() returns TRUE for all four URLs
  3. print(as_completed(future_to_page)) gives a generator object
  4. print('future =', future) shows that all four items in the for loop return have raised BrokenProcessPool. ("future =
  5. Doing everything before calling future.result() throws no errors

So how do I make this ProcessPool work?

0

There are 0 best solutions below