Can instances of classes defined in a module (i.e. not top level) be pickled somehow?

Question

According to the documentation, this is not possible. But I am wondering if there is a workaround to this.

It seems like a very arbitrary limitation that would mess up with the overall code structure

I tried temporarily moving my class to the top level and it worked as expected.

I tried following "solution 3" in the link below but that didn't seem to work: https://copyprogramming.com/howto/i-can-pickle-local-objects-if-i-use-a-derived-class

I have the following folder structure:

MainModule.py
libraries
 |- class_A.py
 |- processor.py

__main__ in processor.py is executed to create an instance of class_A (with data) and pickle it.

Code in processor.py

import dill as pickle
from class_A import Class_A

data = 4
instance_of_A = Class_A(data)

path: str = r"C:\Users\ErikJ.GiesenLoo\Desktop\2023\EvdVds\instance_A.pkl"
with open(path, "wb") as file:
    pickle.dump(instance_of_A, file)

Code in class_A

class Class_A:
    data = 0
    def __init__(self, data):
        self.data = data

Code in MainModule:

import dill as pickle
from libraries.class_A import Class_A


path: str = r"C:\Users\ErikJ.GiesenLoo\Desktop\2023\EvdVds\instance_A.pkl"
with open(path, "rb") as file:
    instance_A = pickle.load(file)

    print(instance_A.data)

MainModule.py then tries to unpickle the saved data, but unpickling only works if I move class_A.py outside of the libraries folder.

Error:

ModuleNotFoundError: No module named 'class_A'

Answer 1

So you are misunderstanding what the documentation is saying. By "not at the top level" it doesn't mean a submodule, it means something like:

def make_class():
    class A:
        pass
    return A

A = make_class()
a = A()

Although, I believe dill may be able to handle this, but definitely, the default pickle will not.

The issue you are encountering is that aren't packaging your project, and relying on the the default behavior of the directory of the script script being added to the module search path.

libraries.class_a.Class_A is not the same as class_a.Class_A. When you pickle your class, it is being recognized as class_A.Class_A, when you try to unpickle it, there is no such class, only libraries.class_A.Class_A.

You should package and install (perhaps in editable mode) your project. Then you should use imports consistently.

The laziest approach is to explicitly set the PYTHONPATH.

Change the import in processor.py to:

 from libraries.class_A import Class_A

Then, supposing your working directory is the directory that contains MainModule.py, you should do:

PYTHONPATH=. python libraries/processor.py

To execute that script, and you should do:

PYTHONPATH=. python MainModule.py

To execute the main script (although that isn't really necessary, it is just for consistency's sake).