Passing in a dictionary as one argument of many using pool.starmap()

Question

I'm trying to use pool.starmap() to make many concurrent post requests and URL retrievals. The function that I've built, let's call it "download", takes 4 arguments, 1 of them being a JSON payload. This function downloads a file and returns the path to that file.

Right now I have those arguments as a list of tuples, but pool.starmap() isn't playing nice with the JSON payload, which is just a dictionary. it seems to try to pass in items of that dictionary as arguments, so in my case raises this exception:

"download() takes 4 positional arguments but 27 were given"

dataList = [(downloadFolder, jsonPayload, arg3, arg4), (downloadFolder, jsonPayload, arg3, arg4)]

if __name__ == "__main__":
    with Pool(3) as pool:
        filePaths = [pool.starmap(download, i) for i in dataList]

Does anybody know away of passing in a dictionary as one argument of many? I've already thought about changing my request to requests.post(URL, data = x) instead of requests.post(URL, json = x) but this function is used in a lot of other places already so I'd rather not do that.

Answer 1

You don't need the list comprehension. starmap does that iteration for you.

filePaths = pool.starmap(download, dataList)

Your code is unpacking the dictionaries into separate arguments to download.