I stored some pandas dataframes in a shelve and then stored the same dataframes but potentially containing a bit more data (maybe a few % more). The second shelve is about twice the size of the first one.
Does the shelve module pre-allocate space in chunks by any chance? My first shelve is ~ 100MB the second one is ~ 200MB. The increase in the size of the shelves does not correspond to the increase in the size of the underlying data. Could it be that shelve can only allocate in multiples of 100MB? So if I need 90MB it will allocate 100MB and if I need 110MB, it will allocate 200MB?