I am currently writing a python program that uses up a lot of RAM. I am aware that I could use the garbage collector to "free up" the ram and avoid this problem. However, I am using Numba to speed up my code, which makes the part where I would need to free the RAM incompatible with the garbage collector. Plus, it would slow down my code to the point where it would be totally useless anyway. Right now I am testing the code on a MacBook Pro which has 32 GB of RAM. I know a bit about software developing but not hardware. I am going to translate my code from the Mac onto raspberry pi's and Nano Pi Fire 3's. This is because I am using the multiprocessing module (Raspberry Pi has 4 cores and Nano Pi Fire 3 has 8). I am also obviously using the MPI module for passing messages. One solution I have found to add more RAM after brief searching has been swap files which are fairly easy to set up on the 2 types of boards I am using. However, I want to know if that is actually a viable option for a python code? If I allowed a swap file of 128 GB and utilized zRAM. This is for boards whose sole purpose is to run this code, would I no longer run into a problem with RAM?
This is the reason why I think that it is a problem with the RAM:
The first indication is if I don't speed up a specific function with Numba, then the program runs completely fine, except a little slower. However, when I use the speedup from Numba this is the crash report I receive:
VM Region Summary:
ReadOnly portion of Libraries: Total=688.6M resident=0K(0%) swapped_out_or_unallocated=688.6M(100%)
Writable regions: Total=1.0G written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=1.0G(100%)
VIRTUAL REGION
REGION TYPE SIZE COUNT (non-coalesced)
=========== ======= =======
Dispatch continuations 32.0M 1
Kernel Alloc Once 8K 1
MALLOC 750.4M 1263
MALLOC guard page 16K 3
MALLOC_LARGE (reserved) 384K 2 reserved VM address space (unallocated)
STACK GUARD 32K 8
Stack 19.6M 8
VM_ALLOCATE 101.2M 723
VM_ALLOCATE (reserved) 160.0M 2 reserved VM address space (unallocated)
__DATA 28.3M 399
__DATA_CONST 20K 1
__FONT_DATA 4K 1
__LINKEDIT 407.9M 116
__OBJC_RO 32.2M 1
__OBJC_RW 1892K 2
__TEXT 280.7M 359
__UNICODE 564K 1
shared memory 12K 3
=========== ======= =======
TOTAL 1.8G 2894
TOTAL, minus reserved VM space 1.6G 2894
Model: MacBookPro16,1, 8 processors, 8-Core Intel Core i9, 2.3 GHz, 16 GB, SMC
So, as can be seen there is a lot of RAM and swap space being used within my program. If I cannot find a solution or the adding of swap space to the boards does not work then obviously I will have to sacrifice speed for actually working. However, it would be nice if knowing the solution I think will work will actually work before I potentially waste time doing so.