Pickling enum types with cloudpickle

[pcmoritz]

Dear all,

cloudpickle currently doesn't support pickling types of type enum.EnumMeta. Here is an example to reproduce (on the latest release cloudpickle 0.7):

import cloudpickle
from enum import IntEnum

class Color(IntEnum):
    RED = 1

u = cloudpickle.dumps(Color)
cloudpickle.loads(u)

gives the error

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-1-ff791d51315b> in <module>
      6 
      7 u = cloudpickle.dumps(Color)
----> 8 cloudpickle.loads(u)

~/anaconda3/lib/python3.7/enum.py in __new__(metacls, cls, bases, classdict)
    149         # save enum items into separate mapping so they don't get baked into
    150         # the new class
--> 151         enum_members = {k: classdict[k] for k in classdict._member_names}
    152         for name in classdict._member_names:
    153             del classdict[name]

AttributeError: 'dict' object has no attribute '_member_names'

Any help with this is appreciated!

-- Philipp.

[ogrisel]

This is a bug because the same code works with pickle.dumps / pickle.loads.

[pierreglaser]

For this specific case, Enum classes derive from a custom metaclass, enumMeta, that has a __new__ method breaking the classic type.__new__ signature (it requires classdict to be an _EnumDict instead of a dict)

In general, cloudpickle does not currenty provide support for classes with metaclasses that break the type.__new__ signature. We can add a hook for Enum classes and subclasses as they are part of the standard library.

[ogrisel]

By hook you mean a dedicated save_enum method registered in the CloudPickler.dispatch registry?

[pierreglaser]

by hook I mean a special handling in save_dynamic_class when type(obj) is not type, but enum.EnumMeta.

EDIT: better use type.new_class as stated below.

[pierreglaser]

Some updates: EnumMeta breaks type.__new__, but type.__new__ this is not the right way to create classes anyway.
The right way to create new classes is to be PEP 3115 compliant and to use types.new_class. EnumMeta for example is PEP3115 compliant, so implementing this mechanism would solve this issue.

A simple example:

In [1]: import enum                                                                                                                                                                                                                           
                                                                                                                                                                                                                                              
In [2]: import types                                                                                                                                                                                                                          
                                                                                                                                                                                                                                              
In [3]: type(enum.IntEnum)                                                                                                                                                                                                                    
Out[3]: enum.EnumMeta                                                                                                                                                                                                                         
                                                                                                                                                                                                                                              
In [4]: Color = types.new_class('Color', bases=(enum.IntEnum,), kwds=None, exec_body=None)                                                                                                                                                    
                                                                                                                                                                                                                                              
In [5]: type(Color)                                                                                                                                                                                                                           
Out[5]: enum.EnumMeta                                                                                                                                                                                                                         
                                                                                                                                                                                                                                              
In [6]: Color                                                                                                                                                                                                                                 
Out[6]: <enum 'Color'>                                                                                                                                                                                                                        
                                           
In [7]: issubclass(Color, enum.IntEnum)                                                                                                                                                                                                       
Out[7]: True                                                                                                                                                                                                       

[ogrisel]

@pierreglaser actually I could not make types.new_class work because of cyclic references between the Enum subclass and its instances. I have adopted another strategy based on the EnumMeta.__call__ method (the functional API of Enums) in #246.

[pierreglaser]

Should this be closed now that #246 got merged?

[ogrisel]

Indeed.