[talumbau]

JIRA ID : AIHPBLAS-4086

Targets GemmFromAnywhere (#11155) rather than develop, so @jinchen62 can test it directly on that branch.

Motivation

The preliminary(hipblaslt) job on #11155 fails with 127 failed, 5 passed in the tensilelite common tests, while the same tests pass locally on a gfx942 box. The entire 140MB CI log contains exactly one distinct error, repeated once per failed test:

.../caches/4463c0a054b3/source/library/gfx942/TensileLibrary.yaml:55:1: error: unknown key 'solutions'
terminate called after throwing an instance of 'std::runtime_error'
  what():  Failed to load solution library
→ Aborted (core dumped), exit 134

The FileNotFoundError on 00_Final.csv that the test_use_cache tests report is purely downstream — the client aborted, so no benchmark CSV was ever written.

This is not a bug in #11155's logic. It is a pre-existing library-shadowing bug in the test harness that #11155 is simply the first PR to expose, because it is the first to change the solution schema (CustomKernelName → customKernel).

Technical Details

1. tox.ini builds tensilelite-client into build_tmp/ and hands it to pytest via --prebuilt-client.
2. That client links libtensilelite-host.so.1 — where the YAML deserializer (src/llvm/Loading.cpp) lives — and recorded a RUNPATH, not an RPATH.
3. RUNPATH is searched after LD_LIBRARY_PATH. tox.ini sets LD_LIBRARY_PATH = {ROCM_PATH}/lib:..., and the Jenkins job independently exports LD_LIBRARY_PATH=/opt/rocm/lib.
4. tensilelite/CMakeLists.txt:170 installs the tensilelite-host target, so on a CI container with hipBLASLt installed, /opt/rocm/lib/libtensilelite-host.so.1 exists.
5. The freshly built host library is therefore silently bypassed; the client runs against the older installed one, which does not understand the new schema.

Why it cannot fail on a typical dev box: local /opt/rocm/lib has only libhipblaslt.so* and no libtensilelite-host.so, so the lookup falls through to RUNPATH and picks up the correct fresh build.

The fix links the client with -Wl,--disable-new-dtags, emitting RPATH, which is searched before LD_LIBRARY_PATH. tensilelite-client has no install() rule — it is a build/test-only binary — so there is no packaging or install-time impact, and CMAKE_INSTALL_RPATH is untouched.

Test Plan

On a gfx942 box (ROCm 7.2.4), with a stale libtensilelite-host.so.1 built from develop@merge-base placed first on LD_LIBRARY_PATH to emulate the CI container:

1. Direct client check: linker tag, library resolution, and a real run against the exact ClientParameters.ini from a failing cache dir.
2. End-to-end: CI's own pipeline — tox -e py3 with --cxx-flags-release=-O3, sdist install, invoke build-client — scoped to the previously failing use_cache tests.
3. Negative control: revert the flag, relink, re-run with tox's exact LD_LIBRARY_PATH.

Test Result

Direct client check:

	linker tag	resolves to	client exit
before	RUNPATH	stale lib	134, Failed to load solution library
after	RPATH	build-tree lib	0

End-to-end with the stale lib injected: 5 passed (test_use_cache.py::test_use_cache, test_compile_and_benchmark, test_use_cache_creates_cache_dir, test_use_cache_different_params, test_build_only.py::test_use_cache) — all previously failing in CI.

Negative control: with the flag reverted, the client picks the stale library and aborts 134 under tox's exact LD_LIBRARY_PATH. So the green run is attributable to the fix, not to the stale library never being on the path.

Caveats, stated plainly:

• Verification used a develop-built stand-in for the stale library, not the real ROCm-installed one from CI's container (which also runs Clang 24.0.0 vs 22.0.0 locally). The mechanism is proven; the exact CI binary was not reproducible off-box. Consequently my repro reports unknown key 'customKernel' where CI reports unknown key 'solutions' — same failure mode, different key, because the CI container's library is older than develop@merge-base.
• The real confirmation is CI itself: if this hypothesis is right, preliminary(hipblaslt) on GemmFromAnywhere + this commit should go green.
• cpu-gemm-driver still has RUNPATH and the identical latent hazard. Deliberately left alone to keep this change minimal; it is a one-line follow-up.

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/TheRock/blob/main/GOVERNANCE.md#pull-requests.

Risk level

Low. Six lines in one CMakeLists.txt, affecting the link of a single non-installed test binary. The worst case is that LD_LIBRARY_PATH can no longer override the host library for tensilelite-client — which is precisely the behaviour being fixed, but worth noting if anyone deliberately relies on that override to A/B a library build.

🤖 Generated with Claude Code