Issue with n_jobs and pickling?

[jmhessel]

Hi there,

I am trying to create a CDClassifier with n_jobs = -1. However, when I try to do it, I get an error about pickling objects:

TypeError: can't pickle lightning.impl.primal_cd_fast.SquaredHinge objects

Any ideas?

[vene]

I can reproduce. The problem happens not when creating the object but when attempting to fit.

numpy 1.12.1, lightning 0.4, sklearn 0.18.1, freshly installed from conda over Bash for Windows:

Python 3.6.1 |Continuum Analytics, Inc.| (default, May 11 2017, 13:09:58)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from lightning.classification import CDClassifier

In [2]: CDClassifier(n_jobs=-1)
Out[2]:
CDClassifier(C=1.0, Cd=1.0, alpha=1.0, beta=0.5, callback=None,
       debiasing=False, loss='squared_hinge', max_iter=50,
       max_steps='auto', multiclass=False, n_calls=100, n_jobs=-1,
       penalty='l2', permute=True, random_state=None, selection='cyclic',
       shrinking=True, sigma=0.01, termination='violation_sum', tol=0.001,
       verbose=0, warm_debiasing=False, warm_start=False)

In [3]: from sklearn.datasets import load_digits

In [4]: X, y = load_digits(return_X_y=True)

In [5]: clf = CDClassifier(n_jobs=-1)

In [6]: clf.fit(X, y)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-6-1a154f38b9fb> in <module>()
----> 1 clf.fit(X, y)

/mnt/C/conda/lib/python3.6/site-packages/lightning/impl/primal_cd.py in fit(self, X, y)
    317                                         self.verbose)
    318                     for k in xrange(n_vectors))
--> 319             model = Parallel(n_jobs=self.n_jobs, verbose=self.verbose)(jobs)
    320             viol, coefs, errors = zip(*model)
    321             self.coef_ = np.asarray(coefs)

/mnt/C/conda/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py in __call__(self, iterable)
    766                 # consumption.
    767                 self._iterating = False
--> 768             self.retrieve()
    769             # Make sure that we get a last message telling us we are done
    770             elapsed_time = time.time() - self._start_time

/mnt/C/conda/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py in retrieve(self)
    717                     ensure_ready = self._managed_backend
    718                     backend.abort_everything(ensure_ready=ensure_ready)
--> 719                 raise exception
    720
    721     def __call__(self, iterable):

/mnt/C/conda/lib/python3.6/site-packages/sklearn/externals/joblib/parallel.py in retrieve(self)
    680                 # check if timeout supported in backend future implementation
    681                 if 'timeout' in getfullargspec(job.get).args:
--> 682                     self._output.extend(job.get(timeout=self.timeout))
    683                 else:
    684                     self._output.extend(job.get())

/mnt/C/conda/lib/python3.6/multiprocessing/pool.py in get(self, timeout)
    606             return self._value
    607         else:
--> 608             raise self._value
    609
    610     def _set(self, i, obj):

/mnt/C/conda/lib/python3.6/multiprocessing/pool.py in _handle_tasks(taskqueue, put, outqueue, pool, cache)
    383                         break
    384                     try:
--> 385                         put(task)
    386                     except Exception as e:
    387                         job, ind = task[:2]

/mnt/C/conda/lib/python3.6/site-packages/sklearn/externals/joblib/pool.py in send(obj)
    369             def send(obj):
    370                 buffer = BytesIO()
--> 371                 CustomizablePickler(buffer, self._reducers).dump(obj)
    372                 self._writer.send_bytes(buffer.getvalue())
    373             self._send = send

TypeError: can't pickle lightning.impl.primal_cd_fast.SquaredHinge objects

In [7]: clf.set_params(n_jobs=1)
Out[7]:
CDClassifier(C=1.0, Cd=1.0, alpha=1.0, beta=0.5, callback=None,
       debiasing=False, loss='squared_hinge', max_iter=50,
       max_steps='auto', multiclass=False, n_calls=100, n_jobs=1,
       penalty='l2', permute=True, random_state=None, selection='cyclic',
       shrinking=True, sigma=0.01, termination='violation_sum', tol=0.001,
       verbose=0, warm_debiasing=False, warm_start=False)

In [8]: clf.fit(X, y)
Out[8]:
CDClassifier(C=1.0, Cd=1.0, alpha=1.0, beta=0.5, callback=None,
       debiasing=False, loss='squared_hinge', max_iter=50,
       max_steps='auto', multiclass=False, n_calls=100, n_jobs=1,
       penalty='l2', permute=True, random_state=None, selection='cyclic',
       shrinking=True, sigma=0.01, termination='violation_sum', tol=0.001,
       verbose=0, warm_debiasing=False, warm_start=False)

Probably the fix is related to implementing get/set_state

[vene]

Interestingly this only fails in python3.6, not in python3.5!

Gist to reproduce is here

The envs I used can be obtained by:

conda create -n py35 python=3.5
# switch to new env
conda install -c conda-forge sklearn-contrib-lightning
conda install scikit-learn

conda create -n py36  # current at the time of this comment
# switch to new env
conda install -c conda-forge sklearn-contrib-lightning
conda install scikit-learn

[mblondel]

Thanks for the investigation @vene!

[vene]

It appears related to this change in python 3.6:

• bpo-22995: Instances of extension types with a state that aren’t subclasses of list or dict and haven’t implemented any pickle-related methods (__reduce__, __reduce_ex__, __getnewargs__, __getnewargs_ex__, or __getstate__), can no longer be pickled. Including memoryview.