Skip to content

[ROCM-3941] Python packaging rework: flat loader, dist-packages install, wheel build - #7822

Closed
amd-juwillia wants to merge 1 commit into
developfrom
juwillia/py-loader-dist-packages
Closed

amd-juwillia wants to merge 1 commit into
developfrom
juwillia/py-loader-dist-packages

Conversation

@amd-juwillia

@amd-juwillia amd-juwillia commented Jun 25, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

amd-smi shipped its Python module by having the system .deb/.rpm postinst call pip install and drop an amdsmi.pth, with a ~200-line ROCm-path-walking loader. This drops all pip/.pth plumbing, installs the module directly into the system interpreter's site-packages, and adds an opt-in SONAME-isolated wheel build so a PyPI wheel can coexist in-process with the system library.

Why loader + wheel build ship together: develop's amdsmi-manylinux-build.yml (from #5725) runs on every projects/amdsmi/** PR and configures -DBUILD_PYTHON_WHEEL=ON. A loader-only PR would either fail that job (no wheel target) or emit a wheel that silently falls back to a system libamd_smi.so. The complete, correct wheel build therefore belongs in this base PR; CI wiring and tests stack on top (#7823, #7824).

Technical Details

Loader (py-interface/amdsmi_wrapper.py, tools/generator.py — regenerated in lockstep):

  • Flat 3-case ladder: AMDSMI_LIB_OVERRIDE, a bundled libamd_smi_python.so, then system libamd_smi.so; a _MissingLibrary sentinel keeps import tolerant.

Install layout (py-interface/CMakeLists.txt):

  • Query /usr/bin/python3 for its real on-sys.path dir and install(DIRECTORY ...) the module there, excluding libamd_smi*.so.

Packaging (DEBIAN/postinst.in, prerm.in, RPM/post.in, preun.in, CMakeLists.txt):

  • Remove pip install/uninstall + .pth; drop the python3-pip/setuptools/wheel deps and setup.py.in.

Wheel build (src/CMakeLists.txt, py-interface/CMakeLists.txt, tools/disable_system_fallback.py, CMakeLists.txt):

  • BUILD_PYTHON_WHEEL (default OFF, requires BUILD_SHARED_LIBS): a SONAME-isolated libamd_smi_python.so linked -Bsymbolic-functions, and a python_wheel target that flips _AMDSMI_ALLOW_SYSTEM_FALLBACK to False in the staged wrapper so the wheel never loads a system lib.

CLI / build (amdsmi_cli/amdsmi_cli.py, amdsmi_init.py, src/CMakeLists.txt):

  • Pin shebang to /usr/bin/python3; drop a dead guard; strip the system lib in Release.

Split 1 of 3 from #5726. #7823 (CI job) and #7824 (tests) stack on this; all target develop.

JIRA ID

Resolves ROCM-3941

Test Plan

  • python3 projects/amdsmi/tools/build_wheel.py --project-dir projects/amdsmi --no-repair (mirrors the manylinux CI).
  • Multi-OS Docker build/install/verify.

Test Result

  • build_wheel.py builds clean; the wheel bundles only libamd_smi_python.so and ships _AMDSMI_ALLOW_SYSTEM_FALLBACK = False.
  • import amdsmi + amdsmi_init() succeed via /usr/bin/python3 with no .pth/pip across the AMD CI images.

Submission Checklist

…ll, wheel build

- Replace the ROCm-path-walking resolver in amdsmi_wrapper.py (and
  tools/generator.py, regenerated in lockstep) with a flat 3-case loader:
  AMDSMI_LIB_OVERRIDE, a bundled libamd_smi_python.so, then system
  libamd_smi.so; a _MissingLibrary sentinel keeps import tolerant.
- Install the amdsmi module directly into the system python's
  site-packages via install(DIRECTORY) instead of pip and a .pth file;
  drop pip plumbing from the DEBIAN/RPM post/pre scripts and remove the
  python3-pip/setuptools/wheel package dependencies and setup.py.in.
- Add BUILD_PYTHON_WHEEL (default OFF, requires BUILD_SHARED_LIBS): build a
  SONAME-isolated libamd_smi_python.so via -Bsymbolic-functions and a
  python_wheel target that flips the loader's system fallback off with
  tools/disable_system_fallback.py so the wheel never loads a system lib.
- Pin the amdsmi_cli shebang to /usr/bin/python3, drop the dead
  cpu-handles guard, and strip the system library in Release builds.

Signed-off-by: Justin Williams <Justin.Williams@amd.com>
@amd-juwillia
amd-juwillia force-pushed the juwillia/py-loader-dist-packages branch from e91b0f8 to 50e158f Compare June 26, 2026 17:25
@therock-pr-bot

therock-pr-bot Bot commented Jun 26, 2026 •

Copy link
Copy Markdown

❌ PR Check — Action Required

Check Status Details
🌿 Branch Name ❌ Fail Branch name does not match allowed patterns.
Branch: juwillia/py-loader-dist-packages
Allowed patterns:
- ^users\/[A-Za-z0-9][A-Za-z0-9\-]*\/.+
- ^shared\/.+
- ^[A-Za-z0-9][A-Za-z0-9\-_]*$
- ^dependabot\/.+
- ^revert-[0-9]+-.+
📝 PR Title/Description ❌ Fail Error: Title is too long (84 characters).
Expected: at most 80 characters.
Desired format: type(optional-scope): short description
───
Error: Title does not follow Conventional Commits style.
Expected: start with a valid type (feat, fix, docs, …).
Desired format: type(optional-scope): short description
⛔ Forbidden Files ✅ Pass —
🧪 Unit Test ❌ Fail Error: Source/code files changed without an accompanying unit test.
Expected: add at least one test file named like test_<name>.py / test_<name>.cpp (or <name>_test.*).
Current: code file(s) changed: projects/amdsmi/amdsmi_cli/amdsmi_cli.py, projects/amdsmi/amdsmi_cli/amdsmi_init.py, projects/amdsmi/py-interface/amdsmi_wrapper.py, projects/amdsmi/tools/disable_system_fallback.py, projects/amdsmi/tools/generator.py; no test file found
🔎 pre-commit ⏳ Pending ⏳ Still running…
🚫 Draft PR 🔜 To Be Enabled —
🚩 Feature Flag 🔜 To Be Enabled —
📊 Code Coverage 🔜 To Be Enabled —

⚠️ 3 policy check(s) failed. Please address the issues above before this PR can be Reviewed.

🚫 Please fix the failed policies

  • ❌ Branch Name
  • ❌ PR Title/Description
  • ❌ Unit Test

The Not ready to Review label was added to this PR. Once all policies pass, the label is removed automatically.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

@therock-pr-bot

Copy link
Copy Markdown

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

  • ❌ Branch Name
  • ❌ PR Title/Description
  • ❌ Unit Test

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

@amd-juwillia amd-juwillia changed the title [ROCM-3941] Flat loader and dist-packages install, drop pip/.pth [ROCM-3941] Python packaging rework: flat loader, dist-packages install, wheel build Jun 26, 2026
@amd-juwillia
amd-juwillia deleted the juwillia/py-loader-dist-packages branch June 28, 2026 01:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant