I have multiple huge files (around 10-12 files, 1-2Gb each) that need to be downloaded every hour. I store them as a Hashmap, where the key is the file version, and the value is the file contents in list of strings. The problem we face is that every hour when new files are downloaded, the GC kicks in and cleans up the old files, which causes long pauses for the system, since the files are huge. I am thinking of a solution where we can store these file contents off-heap. For this, we explored Chronicle Map.
Question 1: While downloading, does OS use any on-heap buffers or data structures? If yes, then it does not matter if I store the file off heap, since on-heap memory has already been allocated. Is there a way to download and store files off-heap, while using no on-heap memory at all for storing the file?
Question 2: is there a way to store files off heap, and just keep a reference of this memory as a value in my hashmap, thus avoiding the use of any special data structure like Chronicle map.