Skip to content

[rhoai-2.25] chore: cherry-pick developer tooling QoL from main #2056

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jiridanek wants to merge 3 commits into
base: rhoai-2.25
Choose a base branch
from
Open

[rhoai-2.25] chore: cherry-pick developer tooling QoL from main #2056

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f83766c
fix(CVE): centralize CVE constraints in dependencies/cve-constraints.…
jiridanek Feb 3, 2026
a227f72
ISSUE #3036: chore(scripts/): rewrite `scripts/pylocks_generator.sh` …
jiridanek Mar 5, 2026
98dfacd
chore(ci): add Python 3.14 setup to workflows to avoid older uv issues
jiridanek Mar 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .github/workflows/build-notebooks-TEMPLATE.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -246,13 +246,14 @@ jobs:
uses: astral-sh/setup-uv@v7
with:
version-file: uv.toml
python-version-file: .python-version
enable-cache: true
cache-dependency-glob: "uv.lock"

- name: Install Python
uses: actions/setup-python@v6
with:
python-version: "3.14"
python-version-file: ".python-version"

- name: Check uv is installed correctly
run: uv version
Expand Down
7 changes: 4 additions & 3 deletions .github/workflows/code-quality.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,14 @@ jobs:
uses: astral-sh/setup-uv@v7
with:
version-file: uv.toml
python-version-file: .python-version
enable-cache: true
cache-dependency-glob: "uv.lock"

- name: Install Python
uses: actions/setup-python@v6
with:
python-version: "3.14"
python-version-file: ".python-version"

- name: Rerun all code generators we have
run: bash ci/generate_code.sh
Expand All @@ -52,7 +53,7 @@ jobs:
uses: astral-sh/setup-uv@v7
with:
version-file: uv.toml
python-version: "3.14"
python-version-file: .python-version
enable-cache: true
cache-dependency-glob: "uv.lock"

Expand Down
5 changes: 3 additions & 2 deletions .github/workflows/docs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,14 @@ jobs:
uses: astral-sh/setup-uv@v7
with:
version-file: uv.toml
python-version-file: .python-version
enable-cache: true
cache-dependency-glob: "uv.lock"

- name: Install Python
uses: actions/setup-python@v6
with:
python-version: "3.14"
python-version-file: ".python-version"

- name: Run the release notes script
run: |
Expand Down
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ repos:
- id: pyright
name: Run Pyright on all files
# entry: /bin/bash -c 'find. -name "*.py" | xargs pyright --pythonversion 3.12'
entry: ./uv run pyright --pythonversion 3.14
entry: ./uv run pyright --pythonversion 3.12
pass_filenames: true
types_or: [python, pyi]
language: system
Expand Down
2 changes: 1 addition & 1 deletion .python-version
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
3.14
3.12
133 changes: 47 additions & 86 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ SHELL := bash
.DELETE_ON_ERROR:
MAKEFLAGS += --warn-undefined-variables
MAKEFLAGS += --no-builtin-rules
# Used where we need an empty expansion (avoids undefined variable warning).
empty :=

# todo: leave the default recipe prefix for now
ifeq ($(origin .RECIPEPREFIX), undefined)
Expand All @@ -25,6 +27,11 @@ RELEASE_PYTHON_VERSION ?= 3.12
CONTAINER_BUILD_CACHE_ARGS ?= --no-cache
# whether to push the images to a registry as they are built
PUSH_IMAGES ?= yes
# INDEX_MODE: auto (default), public-index, or rh-index - controls lock file generation
INDEX_MODE ?= auto
# KONFLUX: whether to build images from Dockerfile.konflux.* (default: no)
KONFLUX ?= no


# OS dependant: Generate date, select appropriate cmd to locate container engine
ifdef OS
Expand Down Expand Up @@ -65,24 +72,35 @@ endif
# Build function for the notebook image:
# ARG 1: Image tag name.
# ARG 2: Path of Dockerfile we want to build.
# ARG 3: Path of the build-args conf file to use.
define build_image
$(eval IMAGE_NAME := $(IMAGE_REGISTRY):$(1)-$(IMAGE_TAG))

# Checks if there’s a build-args/*.conf matching the Dockerfile
$(eval BUILD_DIR := $(dir $(2)))
$(eval DOCKERFILE_NAME := $(notdir $(2)))
$(eval CONF_FILE := $(BUILD_DIR)build-args/$(shell echo $(DOCKERFILE_NAME) | cut -d. -f2).conf)
$(eval CONF_FILE := $(3))

# if the conf file exists, transform it into --build-arg KEY=VALUE flags
$(eval BUILD_ARGS := $(shell \
if [ -f $(CONF_FILE) ]; then \
awk -F= '!/^#/ && NF {gsub(/^[ \t]+|[ \t]+$$/, "", $$1); gsub(/^[ \t]+|[ \t]+$$/, "", $$2); printf "--build-arg %s=%s ", $$1, $$2}' $(CONF_FILE); \
fi))

# Make is only used for local and GHA builds (Konflux does not run make).
# Default to local build for hermetic-capable targets: always set LOCAL_BUILD=true
# when prefetch-input/ exists; mount cachi2/output only when it exists (after prefetch).
$(eval LOCAL_BUILD_ARG := $(if $(wildcard $(BUILD_DIR)prefetch-input),--build-arg LOCAL_BUILD=true,))
$(eval CACHI2_VOLUME := $(if $(and $(wildcard cachi2/output),$(wildcard $(BUILD_DIR)prefetch-input)),--volume $(ROOT_DIR)cachi2/output:/cachi2/output:Z,))

$(info # Building $(IMAGE_NAME) using $(DOCKERFILE_NAME) with $(CONF_FILE) and $(BUILD_ARGS)...)

@if [ -d '$(BUILD_DIR)prefetch-input' ] && [ ! -d cachi2/output ]; then \
echo "Prefetch required for hermetic build. Run: scripts/lockfile-generators/prefetch-all.sh --component-dir $(patsubst %/,%,$(BUILD_DIR)) -- see scripts/lockfile-generators/README.md"; \
exit 1; \
Comment on lines +93 to +100
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Mar 27, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Resolve prefetch and cachi2 paths from $(ROOT_DIR).

These checks are currently relative to the caller's CWD, while the rest of the build logic is rooted via ROOT_DIR. Invoking the Makefile outside the repo root can miss an existing cache and incorrectly fail or skip the hermetic mount.

💡 Suggested fix 
-	$(eval LOCAL_BUILD_ARG := $(if $(wildcard $(BUILD_DIR)prefetch-input),--build-arg LOCAL_BUILD=true,))
-	$(eval CACHI2_VOLUME := $(if $(and $(wildcard cachi2/output),$(wildcard $(BUILD_DIR)prefetch-input)),--volume $(ROOT_DIR)cachi2/output:/cachi2/output:Z,))
+	$(eval LOCAL_BUILD_ARG := $(if $(wildcard $(ROOT_DIR)$(BUILD_DIR)prefetch-input),--build-arg LOCAL_BUILD=true,))
+	$(eval CACHI2_VOLUME := $(if $(and $(wildcard $(ROOT_DIR)cachi2/output),$(wildcard $(ROOT_DIR)$(BUILD_DIR)prefetch-input)),--volume $(ROOT_DIR)cachi2/output:/cachi2/output:Z,))
@@
-	`@if` [ -d '$(BUILD_DIR)prefetch-input' ] && [ ! -d cachi2/output ]; then \
+	`@if` [ -d '$(ROOT_DIR)$(BUILD_DIR)prefetch-input' ] && [ ! -d '$(ROOT_DIR)cachi2/output' ]; then \
 	  echo "Prefetch required for hermetic build. Run: scripts/lockfile-generators/prefetch-all.sh --component-dir $(patsubst %/,%,$(BUILD_DIR)) -- see scripts/lockfile-generators/README.md"; \
 	  exit 1; \
 	fi
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@Makefile` around lines 93 - 100, The prefetch and cachi2 existence checks use
caller CWD; update all path checks and wildcards to resolve via ROOT_DIR so they
mirror the rest of the build logic: change uses of $(wildcard
$(BUILD_DIR)prefetch-input), [ -d '$(BUILD_DIR)prefetch-input' ], and [ ! -d
cachi2/output ] to reference $(ROOT_DIR) (e.g.,
$(ROOT_DIR)$(BUILD_DIR)prefetch-input and $(ROOT_DIR)cachi2/output) and likewise
update the CACHI2_VOLUME and LOCAL_BUILD_ARG evaluations to base their wildcard
tests on $(ROOT_DIR) to ensure invocations outside repo root see the same
cache/mount behavior.

Copy link
Copy Markdown
Member Author

@jiridanek jiridanek Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should not be cherrypicked?

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jiridanek, that's a fair point. If the LOCAL_BUILD_ARG / CACHI2_VOLUME / prefetch-input guard block was not intentionally included in this cherry-pick (it doesn't appear in the listed PR objectives for rhoai-2.25), then my path-resolution comment doesn't apply here.

Would you like to:

  1. Revert those specific lines from the build_image function to keep this PR scoped to the intended changes (uv pinning, cve-constraints, pylocks_generator rewrite, Makefile target rename)?
  2. Or keep the changes but address the ROOT_DIR path concern as a follow-up?

If the prefetch/hermetic-build support is indeed out of scope for rhoai-2.25, I can resolve this comment.

🧠 Learnings used 
Learnt from: CR
Repo: red-hat-data-services/notebooks PR: 0
File: AGENTS.md:0-0
Timestamp: 2026-03-26T19:41:58.522Z
Learning: Applies to Makefile : Set `KONFLUX` Makefile variable consistently across build and test steps; use `KONFLUX=no` for `Dockerfile.*` and `manifests/odh/base/` manifests, or `KONFLUX=yes` for `Dockerfile.konflux.*` and `manifests/rhoai/base/` manifests

Learnt from: CR
Repo: red-hat-data-services/notebooks PR: 0
File: AGENTS.md:0-0
Timestamp: 2026-03-26T19:41:58.522Z
Learning: Applies to **/Dockerfile{,.konflux*} : Minimize Docker layers in Dockerfiles

Learnt from: CR
Repo: red-hat-data-services/notebooks PR: 0
File: AGENTS.md:0-0
Timestamp: 2026-03-26T19:41:58.522Z
Learning: Applies to **/Dockerfile{,.konflux*} : Follow security best practices in Dockerfiles

fi
$(ROOT_DIR)/scripts/sandbox.py --dockerfile '$(2)' --platform '$(BUILD_ARCH)' -- \
$(CONTAINER_ENGINE) build $(CONTAINER_BUILD_CACHE_ARGS) --platform=$(BUILD_ARCH) --label release=$(RELEASE) --tag $(IMAGE_NAME) --file '$(2)' $(BUILD_ARGS) {}\;
$(CONTAINER_ENGINE) build $(CONTAINER_BUILD_CACHE_ARGS) $(LOCAL_BUILD_ARG) $(CACHI2_VOLUME) --platform=$(BUILD_ARCH) --label release=$(RELEASE) --tag $(IMAGE_NAME) --file '$(2)' $(BUILD_ARGS) {}\;
endef

# Push function for the notebook image:
Expand All @@ -99,11 +117,14 @@ endef
#
# PUSH_IMAGES: allows skipping podman push
define image
$(info #*# Image build Dockerfile: <$(2)> #(MACHINE-PARSED LINE)#*#...)
$(eval BUILD_DIRECTORY := $(shell echo $(2) | sed 's/\/Dockerfile.*//'))
$(eval VARIANT := $(shell echo $(notdir $(2)) | cut -d. -f2))
$(eval DOCKERFILE := $(BUILD_DIRECTORY)/Dockerfile$(if $(KONFLUX:no=),.konflux,$(empty)).$(VARIANT))
$(eval CONF_FILE := $(BUILD_DIRECTORY)/build-args/$(if $(KONFLUX:no=),konflux.,$(empty))$(shell echo $(VARIANT)).conf)
$(info #*# Image build Dockerfile: <$(DOCKERFILE)> #(MACHINE-PARSED LINE)#*#...)
$(info #*# Image build directory: <$(BUILD_DIRECTORY)> #(MACHINE-PARSED LINE)#*#...)

$(call build_image,$(1),$(2))
$(call build_image,$(1),$(DOCKERFILE),$(CONF_FILE))

$(if $(PUSH_IMAGES:no=),
$(call push_image,$(1))
Expand All @@ -113,7 +134,7 @@ endef
####################################### Build helpers #######################################

# https://stackoverflow.com/questions/78899903/how-to-create-a-make-target-which-is-an-implicit-dependency-for-all-other-target
skip-init-for := all-images deploy% undeploy% test% validate% refresh-pipfilelock-files scan-image-vulnerabilities print-release
skip-init-for := all-images deploy% undeploy% test% validate% refresh-lock-files scan-image-vulnerabilities print-release
ifneq (,$(filter-out $(skip-init-for),$(MAKECMDGOALS) $(.DEFAULT_GOAL)))
$(SELF): bin/buildinputs
endif
Expand Down Expand Up @@ -406,74 +427,20 @@ validate-rstudio-image: bin/kubectl
continue
fi

# This recipe used mainly from the Pipfile.locks Renewal Action
# Default Python version
PYTHON_VERSION ?= 3.12
ROOT_DIR := $(shell pwd)
ifeq ($(PYTHON_VERSION), 3.11)
BASE_DIRS := \
rstudio/rhel9-python-$(PYTHON_VERSION) \
rstudio/c9s-python-$(PYTHON_VERSION)
else ifeq ($(PYTHON_VERSION), 3.12)
BASE_DIRS := \
jupyter/minimal/ubi9-python-$(PYTHON_VERSION) \
jupyter/datascience/ubi9-python-$(PYTHON_VERSION) \
jupyter/pytorch/ubi9-python-$(PYTHON_VERSION) \
jupyter/tensorflow/ubi9-python-$(PYTHON_VERSION) \
jupyter/trustyai/ubi9-python-$(PYTHON_VERSION) \
jupyter/rocm/pytorch/ubi9-python-$(PYTHON_VERSION) \
jupyter/pytorch+llmcompressor/ubi9-python-$(PYTHON_VERSION) \
codeserver/ubi9-python-$(PYTHON_VERSION) \
runtimes/minimal/ubi9-python-$(PYTHON_VERSION) \
runtimes/datascience/ubi9-python-$(PYTHON_VERSION) \
runtimes/pytorch/ubi9-python-$(PYTHON_VERSION) \
runtimes/tensorflow/ubi9-python-$(PYTHON_VERSION) \
runtimes/rocm-pytorch/ubi9-python-$(PYTHON_VERSION) \
runtimes/pytorch+llmcompressor/ubi9-python-$(PYTHON_VERSION) \
runtimes/rocm-tensorflow/ubi9-python-$(PYTHON_VERSION) \
jupyter/rocm/tensorflow/ubi9-python-$(PYTHON_VERSION)
# rstudio/rhel9-python-$(PYTHON_VERSION)
# rstudio/c9s-python-$(PYTHON_VERSION)
else
$(error Invalid Python version $(PYTHON_VERSION))
endif

# Default value is false, can be overridden
# The below directories are not supported on tier-1
INCLUDE_OPT_DIRS ?= false
OPT_DIRS :=

# This recipe gets args, can be used like
# make refresh-pipfilelock-files PYTHON_VERSION=3.11 INCLUDE_OPT_DIRS=false
.PHONY: refresh-pipfilelock-files
refresh-pipfilelock-files:
@echo "Updating Pipfile.lock files for Python $(PYTHON_VERSION)"
@if [ "$(INCLUDE_OPT_DIRS)" = "true" ]; then
echo "Including optional directories"
DIRS="$(BASE_DIRS) $(OPT_DIRS)"
else
DIRS="$(BASE_DIRS)"
fi
for dir in $$DIRS; do
echo "Processing directory: $$dir"
cd $(ROOT_DIR)
if [ -d "$$dir" ]; then
echo "Updating $(PYTHON_VERSION) uv.lock in $$dir"
cd $$dir
if [ -f "pyproject.toml" ]; then
$(ROOT_DIR)/uv lock && rm uv.lock
else
echo "No pyproject.toml found in $$dir, skipping."
fi
else
echo "Skipping $$dir as it does not exist"
fi
done

echo "Regenerating requirements.txt files"
pushd $(ROOT_DIR)
bash $(ROOT_DIR)/scripts/sync-python-lockfiles.sh
popd
# ======================================================================================
# Refresh lock files
# Usage examples:
# gmake refresh-lock-files <- auto mode (rh-index if uv.lock.d/ exists, else public-index)
# gmake refresh-lock-files INDEX_MODE=public-index <- force public-index
# gmake refresh-lock-files INDEX_MODE=public-index DIR=jupyter/minimal/ubi9-python-3.12
# ======================================================================================
DIR ?=
.PHONY: refresh-lock-files
refresh-lock-files:
@echo "==================================================================="
@echo "🔁 Refreshing lock files using INDEX_MODE=$(INDEX_MODE)"
@echo "==================================================================="
@cd $(ROOT_DIR) && ./uv run scripts/pylocks_generator.py $(INDEX_MODE) $(DIR)

# This is only for the workflow action
# For running manually, set the required environment variables
Expand All @@ -483,13 +450,7 @@ scan-image-vulnerabilities:

# This is used primarily for gen_gha_matrix_jobs.py to we know the set of all possible images we may want to build
.PHONY: all-images
ifeq ($(RELEASE_PYTHON_VERSION), 3.11)
all-images: \
rstudio-c9s-python-$(RELEASE_PYTHON_VERSION) \
rstudio-rhel9-python-$(RELEASE_PYTHON_VERSION) \
cuda-rstudio-c9s-python-$(RELEASE_PYTHON_VERSION) \
cuda-rstudio-rhel9-python-$(RELEASE_PYTHON_VERSION)
else ifeq ($(RELEASE_PYTHON_VERSION), 3.12)
ifeq ($(RELEASE_PYTHON_VERSION), 3.12)
all-images: \
jupyter-minimal-ubi9-python-$(RELEASE_PYTHON_VERSION) \
jupyter-datascience-ubi9-python-$(RELEASE_PYTHON_VERSION) \
Expand All @@ -508,12 +469,11 @@ all-images: \
rocm-jupyter-pytorch-ubi9-python-$(RELEASE_PYTHON_VERSION) \
rocm-runtime-pytorch-ubi9-python-$(RELEASE_PYTHON_VERSION) \
rocm-runtime-tensorflow-ubi9-python-$(RELEASE_PYTHON_VERSION) \
rocm-jupyter-tensorflow-ubi9-python-$(RELEASE_PYTHON_VERSION)
# rstudio-c9s-python-$(RELEASE_PYTHON_VERSION)
# cuda-rstudio-c9s-python-$(RELEASE_PYTHON_VERSION)
# rstudio-rhel9-python-$(RELEASE_PYTHON_VERSION)
# cuda-rstudio-rhel9-python-$(RELEASE_PYTHON_VERSION)

rocm-jupyter-tensorflow-ubi9-python-$(RELEASE_PYTHON_VERSION) \
rstudio-c9s-python-$(RELEASE_PYTHON_VERSION) \
cuda-rstudio-c9s-python-$(RELEASE_PYTHON_VERSION) \
rstudio-rhel9-python-$(RELEASE_PYTHON_VERSION) \
cuda-rstudio-rhel9-python-$(RELEASE_PYTHON_VERSION)
else
$(error Invalid Python version $(RELEASE_PYTHON_VERSION))
endif
Expand All @@ -527,3 +487,4 @@ print-release:
test:
@echo "Running quick static tests"
./uv run pytest -m 'not buildonlytest'
@./scripts/check_dockerfile_alignment.sh
10 changes: 5 additions & 5 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,22 +81,22 @@ brew install python@3.14 uv
<summary>Alternatives to <code>./uv</code></summary>

The `./uv` wrapper is the recommended way, but you can also
(replace `0.10.6` below with the version from `uv.toml`):
(replace `0.8.12` below with the version from `uv.toml`):

- **Use `uvx` directly** with an explicit version:
```shell
uvx uv@0.10.6 sync --locked
uvx uv@0.8.12 sync --locked
```
- **Use `uv tool run`** (equivalent, longer form):
```shell
uv tool run uv@0.10.6 sync --locked
uv tool run uv@0.8.12 sync --locked
```
- **Install the exact version** so `uv` works directly:
```shell
# Standalone installer (any OS)
curl -LsSf https://astral.sh/uv/0.10.6/install.sh | sh
curl -LsSf https://astral.sh/uv/0.8.12/install.sh | sh
# Or with pip
pip install uv==0.10.6
pip install uv==0.8.12
```

If your system uv matches the pinned version, you can use `uv` directly —
Expand Down
5 changes: 3 additions & 2 deletions ci/check-software-versions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -255,8 +255,9 @@ def process_tag(tag):
if stop_and_remove_container(container_id) != 0:
log.error(f"Failed to stop/remove the container '{container_id}' for the '{image_ref}' tag!")
print_delimiter()
return 1 # noqa: B012 `return` inside `finally` blocks cause exceptions to be silenced
print_delimiter()
ret_code = 1
else:
print_delimiter()

return ret_code

Expand Down
3 changes: 1 addition & 2 deletions ci/generate_code.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,5 +7,4 @@ REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
uv --version || pip install "uv==0.8.12"

"${REPO_ROOT}/uv" run scripts/dockerfile_fragments.py
"${REPO_ROOT}/uv" run manifests/tools/generate_kustomization.py
bash scripts/pylocks_generator.sh
"${REPO_ROOT}/uv" run scripts/pylocks_generator.py
9 changes: 7 additions & 2 deletions dependencies/cve-constraints.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,16 @@
# CVE-induced minimum version constraints
# This file is used with `uv pip compile --constraints`
# This file is used with `uv pip compile --constraints` or UV_CONSTRAINT env var
#
# Format: requirements.txt style (package>=version)
#
# When adding a new CVE fix:
# 1. Add the constraint below with CVE ID and issue reference in a comment
# 2. Regenerate the affected lock files
# 2. Regenerate all lock files with: make refresh-lock-files
# 3. The constraint applies to ALL images during resolution

# RHAIENG-2458: CVE-2025-66418 urllib3 decompression vulnerability
# Upstream: https://github.com/elyra-ai/elyra/issues/3325
urllib3>=2.6.0

# RHAIENG-4014: CVE-2026-33236 NLTK path traversal in XML index files
# Reference: https://access.redhat.com/security/cve/CVE-2026-33236
Expand Down
Loading
Loading