combinations for large n in python

90 Views Asked by At

I'm trying to calculate the combinations of the number 62977 in pairs, and then add this as a new column to my dask data frame. unfortunately I get a memory error even when using delayed function in dask and I don't know how to find a more efficient way. I have to process all of these combinations.

I tried this so far:

da.from_delayed(delayed(lambda: np.array(list(combinations(range(file_len), 2)))),shape=(len(list(combinations(range(file_len), 2))), 2), dtype=int, meta=('comb', int))

file_len=62977

and i get:

Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2022.3.1\plugins\python-ce\helpers\pydev\_pydevd_bundle\pydevd_exec2.py", line 3, in Exec
    exec(exp, global_vars, local_vars)
  File "<input>", line 1, in <module>
MemoryError

is there another way?

Thank you.

0

There are 0 best solutions below