Pickling of dynamic class definition

Question

I am trying to pickle a dynamically generated class as a factory for an alternative class. Something like the following:

import sys, pickle

class BC(object):
    pass

C = type("NewClassName", (BC,), {})

pickle.dump(C, sys.stdout)

This leads to the following error:

pickle.PicklingError: Can't pickle <class '__main__.NewClassName'>: it's not found as __main__.NewClassName

For pickling an object of a dynamically generated class, you can define an __reduce__ method, but is there a way to achieve this only for a class definition.

I don't want to use BC directly, because I only need it as a factory for new classes.

Answer 1

A simple workaround to the error is to use the class name as variable name so that pickle can find it:

import sys, pickle

class BC(object):
    pass

NewClassName = type("NewClassName", (BC,), {})

pickle.dump(NewClassName, sys.stdout)

---

However, this probably doesn't really do what you want. When loading the pickled class:

pickle.loads("""c__main__
NewClassName
p0
.""")

You again get the error:

AttributeError: 'module' object has no attribute 'NewClassName'

unless you've already defined the class.

---

As the documentation states:

pickle can save and restore class instances transparently, however the class definition must be importable and live in the same module as when the object was stored.

So you can't use it to generate new classes, just to make sure your objects refer to the correct classes.

---

There are workarounds like pickling type parameters as shown in the other answer, but even then you will not be able to pickle objects of those dynamic classes without exposing the class in the global namespace of both the pickling and the unpickling process (i.e. __main__.ClassName must refer to the class).

Therefore, I would rethink the whole dynamic class approach.

Answer 2

Try the following:

C = type("C", (BC,), {})

The class must be a module level variable, with the same name as the type name.

However, pickling a class dynamically generated class like this will not work (see the answer from @otus).

---

The best solution I can think of, is to pickle the arguments to type, and then recreate the class again when you unpickle it.

Pickle:

import sys, pickle

class BC(object):
    pass

args = ("NewClassName", (BC,), {})
C = type(*args)
C._pickle_args = args

pickle.dump(C._pickle_args, sys.stdout)

Unpickle:

type_args = pickle.loads("<pickled string">)
C = type(*args)

Answer 3

You could use dill, which can serialize dynamic class definitions. Then you don't need any workarounds, and you can do exactly what you wanted to do.

Python 2.7.7 (default, Jun  2 2014, 01:33:50) 
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dill
>>>>
>>> class BC(object):
...   pass
... 
>>> c = type("NewClassName", (BC,), {})
>>> _c = dill.dumps(c)    
>>> c2 = dill.loads(_c)
>>> c2
<class '__main__.NewClassName'>
>>> 

Get dill here: https://github.com/uqfoundation