I'm trying to calculate the combinations of the number 62977 in pairs, and then add this as a new column to my dask data frame. unfortunately I get a memory error even when using delayed function in dask and I don't know how to find a more efficient way. I have to process all of these combinations.
I tried this so far:
da.from_delayed(delayed(lambda: np.array(list(combinations(range(file_len), 2)))),shape=(len(list(combinations(range(file_len), 2))), 2), dtype=int, meta=('comb', int))
file_len=62977
and i get:
Traceback (most recent call last):
File "C:\Program Files\JetBrains\PyCharm Community Edition 2022.3.1\plugins\python-ce\helpers\pydev\_pydevd_bundle\pydevd_exec2.py", line 3, in Exec
exec(exp, global_vars, local_vars)
File "<input>", line 1, in <module>
MemoryError
is there another way?
Thank you.