pex creation fails under pants w/ No such file or directory error

[asherf]

23:06:05.36 [INFO] Starting: Resolving 3rdparty/python/constraints.txt
23:06:05.38 [WARN] /home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b2_py38/lib/python3.8/site-packages/pants/base/exception_sink.py:313: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
  process_title=setproctitle.getproctitle(),
23:06:05.38 [ERROR] 1 Exception encountered:
Engine traceback:
  in select
  in `typecheck` goal
  in Typecheck using MyPy
  in pants.backend.python.typecheck.mypy.rules.mypy_typecheck_partition
  in pants.backend.python.util_rules.pex.create_pex
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b2_py38/lib/python3.8/site-packages/pants/engine/process.py", line 241, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Building mypy.pex with 1 requirement: mypy==0.782' failed with exit code 1.
stdout:
stderr:
Failed to spawn a job for DistributionTarget(interpreter=PythonInterpreter('/usr/local/bin/python3.8', PythonIdentity('/usr/local/bin/python3.8', 'cp38', 'cp38', 'manylinux2014_x86_64', (3, 8, 5)))): [Errno 2] No such file or directory: '/home/toolchain/.pex/unzipped_pexes/49b4b32295a60a0a4dfc0659c3a44f93560c4a81/.deps/pex-2.1.16-py2.py3-none-any.whl/pex/__pycache__/variables.cpython-37.pyc.139923989690656'
Traceback (most recent call last):
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b2_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 256, in run
    engine_result = self._run_v2()
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b2_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 168, in _run_v2
    return self._maybe_run_v2_body(goals, poll=False)
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b2_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 185, in _maybe_run_v2_body
    return self.graph_session.run_goal_rules(
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b2_py38/lib/python3.8/site-packages/pants/init/engine_initializer.py", line 125, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b2_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 568, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b2_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 527, in _raise_on_error
    raise ExecutionError(
pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:
Engine traceback:
  in select
  in `typecheck` goal
  in Typecheck using MyPy
  in pants.backend.python.typecheck.mypy.rules.mypy_typecheck_partition
  in pants.backend.python.util_rules.pex.create_pex
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0b2_py38/lib/python3.8/site-packages/pants/engine/process.py", line 241, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Building mypy.pex with 1 requirement: mypy==0.782' failed with exit code 1.
stdout:
stderr:
Failed to spawn a job for DistributionTarget(interpreter=PythonInterpreter('/usr/local/bin/python3.8', PythonIdentity('/usr/local/bin/python3.8', 'cp38', 'cp38', 'manylinux2014_x86_64', (3, 8, 5)))): [Errno 2] No such file or directory: '/home/toolchain/.pex/unzipped_pexes/49b4b32295a60a0a4dfc0659c3a44f93560c4a81/.deps/pex-2.1.16-py2.py3-none-any.whl/pex/__pycache__/variables.cpython-37.pyc.139923989690656'

seeing this randomly from time to time in different CI runs.

[stuhood]

Pants now invokes multiple instances of mypy concurrently: it's possible that this is related. But the contents of the __pycache__ should be concurrency safe, afaik.

[jsirois]

Looking very closely at the last line, I think this may be caused by the sys.executable value that lies issue on OSX framework builds (#1009 fixed by #1049) Since what pex thinks is a Python 3.8 interpreter is actually launching a Python 3.7 interpreter (the file not found is a 3.7 bytecode file Failed to spawn a job for DistributionTarget(interpreter=PythonInterpreter('/usr/local/bin/python3.8', PythonIdentity('/usr/local/bin/python3.8', 'cp38', 'cp38', 'manylinux2014_x86_64', (3, 8, 5)))): [Errno 2] No such file or directory: '/home/toolchain/.pex/unzipped_pexes/49b4b32295a60a0a4dfc0659c3a44f93560c4a81/.deps/pex-2.1.16-py2.py3-none-any.whl/pex/__pycache__/variables.cpython-37.pyc.139923989690656'.

I can't provide a causal chain reason yet though so I'll leave this open to work on a bit harder.

[asherf]

this happens in CI (circleci), not OSX.

[jsirois]

Aha, ok - almost certainly something else then. I'll take a look.

[jsirois]

Notes:

1. If this is a straight-up concurrency problem, the relevant 3.8 code is here:
   https://github.com/python/cpython/blob/4c2e299d80c53591f05de2669c0edeaf8acd8544/Lib/importlib/_bootstrap_external.py#L120-L139
   And the relevant 3.7 code is here and the same:
   https://github.com/python/cpython/blob/4e02981de0952f54bf87967f8e10d169d6946b40/Lib/importlib/_bootstrap_external.py#L105-L124
   As commented the code is best effort. Two independent processes could have the same object id for the string representing the path of the final pyc file and could thus race. This seems really unlikely though and I'm not sure how we'd fix this without turning off bytecode compilation.

2. This may not be a straight-up concurrency issue since the interpreter (3.8) vs bytecode path (37) indicates something unexpected.

[jsirois]

Narrowing down a bit more, this error involves running the Pex PEX which is the one released by the Pex proejct and marked as --unzip. This means for the corresponding points above:

1. Since the PEX bootstrap is doing the unzipping into PEX_ROOT/unzipped_pexes using atomic_directory which is UUID4 best-effort it could also eagerly bytecode compile all files while under that lock to circumvent the much weaker best effort of CPython which just relies on object id - aka memory address - uniqueness.
2. This could explain the interpreter mismatch if we're running the Pex PEX with 3.8 but it then re-execs itself with 3.7. That needs more investigation though since the Pex PEX interpreter constraints are ">=2.7,<3.9,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*" which allows for 3.8 and since Pex supposedly always uses the current interpreter instead of re-execing if the current interpreter matches constraints.

[jsirois]

Alright, explanation in hand. The issue here is AtomicDirectory. That code is robust but should only be used when the contents of a directory you want to create atomically are immutable. In the case of --unzip (~/.pex/unzipped_pexes/...) and --not-zip-safe (~/.pex/code/...) and the unpacked wheel cache (~/.pex/installed_wheels/...) where the directory is populated with python code that will be executed later, the directory contents is appended to later by python bytecode compilation. Since AtomicDirectory has at least once semantics, in a race involving more than one process filling the directory atomically simultaneously we can have:

1. 1st process creates the directory.
2. Another process executes against that directory and creates a temporary bytecode file but does not yet rename it: https://github.com/python/cpython/blob/4c2e299d80c53591f05de2669c0edeaf8acd8544/Lib/importlib/_bootstrap_external.py#L126-L132
3. 2nd process (re) creates the directory, losing the temprary bytecode file from 2.
4. The process in 2 finds the temporary bytecode file it wrote is missing when attempting to atomically rename it: https://github.com/python/cpython/blob/4c2e299d80c53591f05de2669c0edeaf8acd8544/Lib/importlib/_bootstrap_external.py#L133

For atomic mutable directory creation we'll need an interprocess lock. Another approach would be to continue with AtomicDirectory but append an identifier to the path unique to the bytecode compilation, say cpython-37. This has the disadvantage though of keeping a copy of source code for each unique interpreter run against it.

[Eric-Arellano]

Do either of these solutions need to be applied unconditionally, or we could try to detect when it's unsafe and only use these fixes then?

[jsirois]

Do either of these solutions need to be applied unconditionally, or we could try to detect when it's unsafe and only use these fixes then?

They need to be applied unconditionally since detecting when its safe amounts to detecting if there are other similar processes racing which amounts to either implementing an interprocess lock (the proposed solution) or else eliminating the need for a lock (the alternative solution).

[asherf]

yay! thanks for fixing @jsirois !

[asherf]

@jsirois need to reopen this... Since I am seeing this with the latest release.

Engine traceback:
  in select
  in pants.core.goals.typecheck.typecheck
  in pants.backend.python.typecheck.mypy.rules.mypy_typecheck
  in pants.backend.python.typecheck.mypy.rules.mypy_typecheck_partition
  in pants.backend.python.util_rules.pex.create_pex
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0rc0_py38/lib/python3.8/site-packages/pants/engine/process.py", line 241, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Resolving 3rdparty/python/constraints.txt' failed with exit code 1.
stdout:

stderr:
Failed to spawn a job for DistributionTarget(interpreter=PythonInterpreter('/usr/local/bin/python3.8', PythonIdentity('/usr/local/bin/python3.8', 'cp38', 'cp38', 'manylinux2014_x86_64', (3, 8, 5)))): [Errno 2] No such file or directory: '/home/toolchain/.pex/unzipped_pexes/d8bfff1518d5d211d4ec95cf18305ff3939eef68/.deps/pex-2.1.18-py2.py3-none-any.whl/pex/__pycache__/platforms.cpython-37.pyc.139727263750640'


Traceback (most recent call last):
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0rc0_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 281, in run
    engine_result = self._run_v2()
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0rc0_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 193, in _run_v2
    return self._maybe_run_v2_body(goals, poll=False)
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0rc0_py38/lib/python3.8/site-packages/pants/bin/local_pants_runner.py", line 210, in _maybe_run_v2_body
    return self.graph_session.run_goal_rules(
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0rc0_py38/lib/python3.8/site-packages/pants/init/engine_initializer.py", line 126, in run_goal_rules
    exit_code = self.scheduler_session.run_goal_rule(
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0rc0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 569, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/home/toolchain/.cache/pants/setup/bootstrap-Linux-x86_64/2.0.0rc0_py38/lib/python3.8/site-packages/pants/engine/internals/scheduler.py", line 528, in _raise_on_error
    raise ExecutionError(
pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:

[jsirois]

Ok. Assuming Linux posix APIs are not broken, which is a very safe bet, the observation of temporary pyc files created by Python can only be via os.walk. Pex has at least one of those.

[jsirois]

Another instance of this was found in #1098 and fixed in #1099.