Clarify builtin function handling in PyPy

[pierreglaser]

Experimental PR to clarify handling of builtin-functions in PyPy.

[codecov]

Codecov Report

Merging #278 into master will decrease coverage by 0.37%.
The diff coverage is 57.14%.

@@            Coverage Diff             @@
##           master     #278      +/-   ##
==========================================
- Coverage   91.58%   91.21%   -0.38%     
==========================================
  Files           1        1              
  Lines         606      603       -3     
  Branches      129      129              
==========================================
- Hits          555      550       -5     
- Misses         31       32       +1     
- Partials       20       21       +1

Impacted Files	Coverage Δ
cloudpickle/cloudpickle.py	91.21% <57.14%> (-0.38%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f00e36e...3547124. Read the comment docs.

[codecov]

Codecov Report

Merging #278 into master will increase coverage by 0.73%.
The diff coverage is 94.44%.

@@            Coverage Diff             @@
##           master     #278      +/-   ##
==========================================
+ Coverage   91.25%   91.99%   +0.73%     
==========================================
  Files           1        1              
  Lines         606      612       +6     
  Branches      129      131       +2     
==========================================
+ Hits          553      563      +10     
+ Misses         33       30       -3     
+ Partials       20       19       -1

Impacted Files	Coverage Δ
cloudpickle/cloudpickle.py	91.99% <94.44%> (+0.73%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e1da949...de9f3aa. Read the comment docs.

[pierreglaser]

This PR comes from a discussion I have IRL with @ogrisel, and by #253 (comment). There are chunks of defensive code, namely:

cloudpickle/cloudpickle/cloudpickle.py

Lines 691 to 697 in f00e36e

	try:
	names = co.co_names
	except AttributeError:
	# PyPy "builtin-code" object
	out_names = set()
	else:
	out_names = {names[oparg] for _, oparg in _walk_global_ops(co)}

and

cloudpickle/cloudpickle/cloudpickle.py

Lines 679 to 682 in f00e36e

	_extract_code_globals_cache = (
	weakref.WeakKeyDictionary()
	if not hasattr(sys, "pypy_version_info")
	else {})

That are supposed to silence edge cases where PyPy builtin functions are treated as dynamic. This case happens because PyPy builtin function __qualname__ support is sometimes flaky.

This makes the code a little bit hard to follow. Also, builtin functions are simpler to pickle than dynamic functions: They do not seem have globals or closure, which limits the risk of reference cycles, so we do not need to use the whole make_skel_func + fill_function armada, but simply rely on save_reduce. All in all, I think they can be pickled separately in a one-liner routine (save_pypy_builtin_func)

As a side note, this new routine is simply us handling PyPy 3.5 bugs. It can be removed when we decide to move on to support 3.6 and later (actually, it can be removed after I fix a further pickling-behavior branching error in classmethod).

@ogrisel WDYT?

[ogrisel]

This is much cleaner. Thanks for this refactoring.

[HyukjinKwon]

Hm, I realised that this breaks lower PyPy (in my case it was PyPy 2.5.1):

from cloudpickle import dumps
from pickle import loads
P = namedtuple("P", "x y")
loads(dumps(P))

Traceback (most recent call last):
  File "/home/jenkins/workspace/SparkPullRequestBuilder@2/python/pyspark/tests/test_serializers.py", line 41, in test_namedtuple
    P2 = loads(dumps(P))
  File "/usr/lib64/pypy-2.5.1/lib-python/2.7/pickle.py", line 1425, in loads
    return Unpickler(file).load()
  File "/usr/lib64/pypy-2.5.1/lib-python/2.7/pickle.py", line 901, in load
    dispatch[key](self)
  File "/usr/lib64/pypy-2.5.1/lib-python/2.7/pickle.py", line 1260, in load_build
    setstate(state)
ValueError: Wrong arguments to function.__setstate__

@ogrisel do we plan to drop Python 2 and PyPy 2.x soon?