Skip to content

Commit a0d09ef

Browse files
committed
feat: introduce blocklist of unwanted requirements
Commit 25da067 introduced a manually curated fragment for `tree-sitter` to work around shortcomings [1] of pip-compile. Generalize this technique by introducing a declarative blocklist and associated script. This feature is going to be needed by three unrelated features: fixing `pydub`, running `pip-compile.sh` on non-Apple platforms, and exploring the "complex platform dependent versioning path" proposed in PR Aider-AI#3444 [2]. As an added benefit, the blocklist script: - allows comments in the blocklist; and - removes extraneous comments from `requirements.txt`, avoiding misattribution and confusion if a human looks at them. [1]: jazzband/pip-tools#639 (comment) [2]: Aider-AI#3444 (comment)
1 parent af8558b commit a0d09ef

10 files changed

+116
-17
lines changed

requirements.txt

+1-4
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# This file was autogenerated by uv via the following command:
2-
# uv pip compile --no-strip-extras --constraint=requirements/common-constraints.txt --output-file=tmp.requirements.txt requirements/requirements.in
2+
# uv pip compile --no-strip-extras --constraint=requirements/common-constraints.txt --output-file=unfiltered/requirements.txt requirements/requirements.in
33
aiohappyeyeballs==2.5.0
44
# via
55
# -c requirements/common-constraints.txt
@@ -368,9 +368,6 @@ tqdm==4.67.1
368368
# -c requirements/common-constraints.txt
369369
# huggingface-hub
370370
# openai
371-
# via
372-
# -c requirements/common-constraints.txt
373-
# tree-sitter-language-pack
374371
tree-sitter-c-sharp==0.23.1
375372
# via
376373
# -c requirements/common-constraints.txt

requirements/blocklist.in

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
# Blocklist of dependencies.
2+
#
3+
# `pip-compile` does not support scenarios where a Python package
4+
# (e.g. `tree-sitter`) is only available for specific architectures,
5+
# OSes, or Python versions. [1]
6+
# This blocklist works around that limitation.
7+
#
8+
# The requirements listed below are to be excluded from all
9+
# requirements files, and maintained by hand in *.add files using
10+
# environment markers.
11+
#
12+
# [1]: https://github.com/jazzband/pip-tools/issues/639#issuecomment-374316252
13+
14+
# See tree-sitter.add
15+
tree-sitter

requirements/common-constraints.txt

+1-3
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# This file was autogenerated by uv via the following command:
2-
# uv pip compile --no-strip-extras --output-file=requirements/common-constraints.txt requirements/requirements.in requirements/requirements-browser.in requirements/requirements-dev.in requirements/requirements-help.in requirements/requirements-playwright.in
2+
# uv pip compile --no-strip-extras --output-file=unfiltered/common-constraints.txt requirements/requirements.in requirements/requirements-browser.in requirements/requirements-dev.in requirements/requirements-help.in requirements/requirements-playwright.in
33
aiohappyeyeballs==2.5.0
44
# via aiohttp
55
aiohttp==3.11.13
@@ -455,8 +455,6 @@ tqdm==4.67.1
455455
# transformers
456456
transformers==4.49.0
457457
# via sentence-transformers
458-
tree-sitter==0.24.0
459-
# via tree-sitter-language-pack
460458
tree-sitter-c-sharp==0.23.1
461459
# via tree-sitter-language-pack
462460
tree-sitter-embedded-template==0.23.2

requirements/requirements-browser.txt

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# This file was autogenerated by uv via the following command:
2-
# uv pip compile --no-strip-extras --constraint=requirements/common-constraints.txt --output-file=requirements/requirements-browser.txt requirements/requirements-browser.in
2+
# uv pip compile --no-strip-extras --constraint=requirements/common-constraints.txt --output-file=unfiltered/requirements-browser.txt requirements/requirements-browser.in
33
altair==5.5.0
44
# via
55
# -c requirements/common-constraints.txt

requirements/requirements-dev.txt

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# This file was autogenerated by uv via the following command:
2-
# uv pip compile --no-strip-extras --constraint=requirements/common-constraints.txt --output-file=requirements/requirements-dev.txt requirements/requirements-dev.in
2+
# uv pip compile --no-strip-extras --constraint=requirements/common-constraints.txt --output-file=unfiltered/requirements-dev.txt requirements/requirements-dev.in
33
build==1.2.2.post1
44
# via
55
# -c requirements/common-constraints.txt

requirements/requirements-help.txt

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# This file was autogenerated by uv via the following command:
2-
# uv pip compile --no-strip-extras --constraint=requirements/common-constraints.txt --output-file=requirements/requirements-help.txt requirements/requirements-help.in
2+
# uv pip compile --no-strip-extras --constraint=requirements/common-constraints.txt --output-file=unfiltered/requirements-help.txt requirements/requirements-help.in
33
aiohappyeyeballs==2.5.0
44
# via
55
# -c requirements/common-constraints.txt

requirements/requirements-playwright.txt

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# This file was autogenerated by uv via the following command:
2-
# uv pip compile --no-strip-extras --constraint=requirements/common-constraints.txt --output-file=requirements/requirements-playwright.txt requirements/requirements-playwright.in
2+
# uv pip compile --no-strip-extras --constraint=requirements/common-constraints.txt --output-file=unfiltered/requirements-playwright.txt requirements/requirements-playwright.in
33
greenlet==3.1.1
44
# via
55
# -c requirements/common-constraints.txt
File renamed without changes.

scripts/block_requirements.py

+73
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
#!/usr/bin/env python
2+
3+
"""Applies a blocklist to requirements files."""
4+
5+
import fileinput
6+
from pathlib import Path
7+
import re
8+
import sys
9+
10+
from packaging.requirements import Requirement
11+
12+
PROJECT_ROOT = Path(__file__).parent.parent.absolute()
13+
BLOCKLIST_PATH = PROJECT_ROOT / "requirements/blocklist.in"
14+
15+
16+
def is_requirement(line):
17+
"""Returns True if the line is an actual requirement."""
18+
19+
def empty():
20+
return not line.strip()
21+
22+
def comment():
23+
return re.match(r"^\s*#", line)
24+
25+
return not empty() and not comment()
26+
27+
28+
def parse_blocklist():
29+
"""Parses a list of blocked requirements."""
30+
with open(BLOCKLIST_PATH, encoding="utf-8") as file:
31+
for line in file:
32+
if is_requirement(line):
33+
yield line.strip()
34+
35+
36+
def parse_requirements():
37+
"""Parses the requirement file(s) given as an input."""
38+
for line in fileinput.input(encoding="utf-8"):
39+
if is_requirement(line):
40+
yield Requirement(line)
41+
42+
43+
def filtered_requirements_lines():
44+
"""Parses the requirement file(s) given as an input.
45+
Filters out all sections that represent a blocked requirement.
46+
"""
47+
include_section = True
48+
blocked_requirements = set(parse_blocklist())
49+
50+
for line in fileinput.input(encoding="utf-8"):
51+
if is_requirement(line):
52+
name = Requirement(line).name
53+
include_section = name not in blocked_requirements
54+
if not include_section:
55+
print(f"Removing blocked requirement: {name}", file=sys.stderr)
56+
if include_section:
57+
yield line
58+
59+
60+
def main():
61+
"""Applies the blocklist to the requirements file(s) given as
62+
an input. Removes blocked requirements.
63+
"""
64+
for line in filtered_requirements_lines():
65+
print(line, end="")
66+
67+
68+
if __name__ == "__main__":
69+
try:
70+
main()
71+
except BrokenPipeError:
72+
# play nice with Unix pipelines
73+
pass

scripts/pip-compile.sh

+22-6
Original file line numberDiff line numberDiff line change
@@ -6,27 +6,34 @@ set -e
66
# Add verbosity flag to see more details about dependency resolution
77
VERBOSITY="-v" # Use -v for less detail, -vvv for even more detail
88

9+
mkdir -p unfiltered
10+
11+
# Clean up temporary directory on exit
12+
trap 'rm -f unfiltered/*.txt; rmdir unfiltered' EXIT
13+
914
# First compile the common constraints of the full requirement suite
1015
# to make sure that all versions are mutually consistent across files
1116
uv pip compile \
1217
$VERBOSITY \
1318
--no-strip-extras \
14-
--output-file=requirements/common-constraints.txt \
19+
--output-file=unfiltered/common-constraints.txt \
1520
requirements/requirements.in \
1621
requirements/requirements-*.in \
1722
$1
23+
scripts/block_requirements.py \
24+
unfiltered/common-constraints.txt \
25+
> requirements/common-constraints.txt
1826

1927
# Compile the base requirements
2028
uv pip compile \
2129
$VERBOSITY \
2230
--no-strip-extras \
2331
--constraint=requirements/common-constraints.txt \
24-
--output-file=tmp.requirements.txt \
32+
--output-file=unfiltered/requirements.txt \
2533
requirements/requirements.in \
2634
$1
27-
28-
grep -v ^tree-sitter= tmp.requirements.txt \
29-
| cat - requirements/tree-sitter.in \
35+
scripts/block_requirements.py unfiltered/requirements.txt \
36+
| cat - requirements/requirements.add/*.txt \
3037
> requirements.txt
3138

3239
# Compile additional requirements files
@@ -37,7 +44,16 @@ for SUFFIX in "${SUFFIXES[@]}"; do
3744
$VERBOSITY \
3845
--no-strip-extras \
3946
--constraint=requirements/common-constraints.txt \
40-
--output-file=requirements/requirements-${SUFFIX}.txt \
47+
--output-file="unfiltered/requirements-${SUFFIX}.txt" \
4148
requirements/requirements-${SUFFIX}.in \
4249
$1
50+
scripts/block_requirements.py \
51+
"unfiltered/requirements-${SUFFIX}.txt" \
52+
| {
53+
if [[ -d "requirements/requirements-${SUFFIX}.add" ]]; then
54+
cat - "requirements/requirements-${SUFFIX}.add"/*.txt
55+
else
56+
cat
57+
fi
58+
} > "requirements/requirements-${SUFFIX}.txt"
4359
done

0 commit comments

Comments
 (0)