Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry on python interpreter launch failures #2278

Merged
merged 1 commit into from
Mar 7, 2024

Conversation

konstin
Copy link
Member

@konstin konstin commented Mar 7, 2024

Sometimes, the first time we read from the stdout of the bytecode compiler python subprocess, we get an empty string back (no newline). If we try to write to stdin, it will often be a broken pipe (#2245). After we got an empty string the first time, we will get the same empty string if we read a line again.

The details of this behavior are mysterious to me, but it seems that it can be identified by the first empty string. We check by inserting starting with a Ready message on the Python side. When we encounter the broken state, we discard the interpreter and try again.

We have to introduce a third timeout check for the interpreter launch itself.

Minimized test script:

#!/usr/bin/env bash

set -euo pipefail

while true; do
  date --iso-8601=seconds # Progress indicator
  rm -rf testenv
  target/profiling/uv venv testenv -q --python 3.12
  VIRTUAL_ENV=$PWD/testenv target/profiling/uv pip install -q --compile wheel==0.42.0
done

Run as

cargo build --profile profiling && bash compile_bug.sh

Fixes #2245

Sometimes, the first time we read from stdout, we get an empty string back (no newline). If
we try to write to stdin, it will often be a broken pipe (#2245). After we got an empty string the first time, we will get the same empty string if we read a line again.

The details of this behavior are mysterious to me, but it seems that it can be identified by the first empty string. We check by inserting starting with a `Ready` message on the Python side. When we encounter the broken state, we discard the interpreter and try again.

Minimized test script:

```bash
#!/usr/bin/env bash

set -euo pipefail

while true; do
  date --iso-8601=seconds # Progress indicator
  rm -rf testenv
  target/profiling/uv venv testenv -q --python 3.12
  VIRTUAL_ENV=$PWD/testenv target/profiling/uv pip install -q --compile wheel==0.42.0
done
```

Run as

```
cargo build --profile profiling && bash compile_bug.sh
```

Fixes #2245
@konstin konstin added the bug Something isn't working label Mar 7, 2024
@konstin konstin merged commit 54311c8 into main Mar 7, 2024
7 checks passed
@konstin konstin deleted the konsti/ignore-python-launch-failures branch March 7, 2024 15:07
@konstin konstin mentioned this pull request Mar 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Sometimes bytecode compilations fails
2 participants