[WIP][LLVM] Test: Validate Auto Shared Library Resolver Correctness.

[SahilPatidar]

This Pull request:

Changes or fixes:

Checklist:

• tested changes locally
• updated the docs (if necessary)

This PR fixes #

[SahilPatidar]

@vgvassilev

[github-actions]

Test Results

0 tests   0 ✅  0s ⏱️
0 suites  0 💤
0 files    0 ❌

Results for commit 5597d85.

♻️ This comment has been updated with latest results.

[SahilPatidar]

Most of these tests are currently failing:

2144: roottest-root-io-evolution-issue-8083-WriteAfterOld  
2145: roottest-root-io-evolution-issue-8083-readfile  
2683: roottest-root-meta-MakeProject-examples  
2684: roottest-root-meta-MakeProject-stltest  
2685: roottest-root-meta-MakeProject-stltest2  
2714: roottest-root-meta-autoloading-ROOT-8432-execcmsWrapper-auto  
2729: roottest-root-meta-autoloading-headerParsingOnDemand-no_autoparse_write  
2730: roottest-root-meta-autoloading-headerParsingOnDemand-no_autoparse_read  
3074: roottest-root-tree-addresses-make  
3138: roottest-root-tree-selectorreader-make  

Some of them are failing with this error:

fatal error: 'libcmswrapper_dictrflx' file not found
R__LOAD_LIBRARY(libcmswrapper_dictrflx)

[vgvassilev]

Most of these tests are currently failing:

2144: roottest-root-io-evolution-issue-8083-WriteAfterOld  
2145: roottest-root-io-evolution-issue-8083-readfile  
2683: roottest-root-meta-MakeProject-examples  
2684: roottest-root-meta-MakeProject-stltest  
2685: roottest-root-meta-MakeProject-stltest2  
2714: roottest-root-meta-autoloading-ROOT-8432-execcmsWrapper-auto  
2729: roottest-root-meta-autoloading-headerParsingOnDemand-no_autoparse_write  
2730: roottest-root-meta-autoloading-headerParsingOnDemand-no_autoparse_read  
3074: roottest-root-tree-addresses-make  
3138: roottest-root-tree-selectorreader-make  

Some of them are failing with this error:

fatal error: 'libcmswrapper_dictrflx' file not found
R__LOAD_LIBRARY(libcmswrapper_dictrflx)

How do we solve that?

[SahilPatidar]

Do you have any idea what this macro is doing? I'm not familiar with it:

2523: In file included from input_line_25:1:
2523: /Users/sahilpatidar/Desktop/root/roottest/root/meta/autoloading/ROOT-8432/execcmsWrapper.C:1:1: fatal error: 'libcmswrapper_dictrflx' file not found
2523: R__LOAD_LIBRARY(libcmswrapper_dictrflx)
2523: ^
2523: /Users/sahilpatidar/Desktop/root/build/include/Rtypes.h:458:35: note: expanded from macro 'R__LOAD_LIBRARY'
2523: # define R__LOAD_LIBRARY(LIBRARY) _R_PragmaStr(cling load ( #LIBRARY ))
2523:                                   ^
2523: /Users/sahilpatidar/Desktop/root/build/include/Rtypes.h:457:26: note: expanded from macro '_R_PragmaStr'
2523: # define _R_PragmaStr(x) _Pragma(#x)
2523:                          ^
2523: <scratch space>:5:40: note: expanded from here
2523:  cling load ( "libcmswrapper_dictrflx" )

It looks like it's trying to load a library into Cling, but I'm not sure exactly where or how this is being triggered.

[vgvassilev]

This is user code forcing a dlopen.

[SahilPatidar]

I'm seeing this error in those tests even on my master branch.

[vgvassilev]

I'm seeing this error in those tests even on my master branch.

You can check here what are the issues https://github.com/root-project/root/runs/47314709783

I see on windows: LLVM Error: Unsupported binary format: C:\ROOT-CI\build\bin\gdk-1.3.dll and then a bunch of:

Error in <AutoloadLibraryMU>: Failed to load library /github/home/ROOT-CI/build/test/stressShapescling::DynamicLibraryManager::loadLibrary(): /github/home/ROOT-CI/build/test/threads: cannot dynamically load position-independent executable

Can we resolve these first?

[SahilPatidar]

LLVM Error: Unsupported binary format: C:\ROOT-CI\build\bin\gdk-1.3.dll
This error occurs because COFF format support for parsing dependencies hasn’t been added yet.
To fix it, we just need to add COFF support, or simply skip it with:

if (Obj->isCOFF())
    return; // TODO: Add COFF support

[vgvassilev]

LLVM Error: Unsupported binary format: C:\ROOT-CI\build\bin\gdk-1.3.dll This error occurs because COFF format support for parsing dependencies hasn’t been added yet. To fix it, we just need to add COFF support, or simply skip it with:

if (Obj->isCOFF())
    return; // TODO: Add COFF support

Sure, if that does not work in the master that’s a reasonable fix.

[SahilPatidar]

Do you think these test failures are related to our changes? I have some async workarounds locally, so we could try them there too.

[vgvassilev]

Do you think these test failures are related to our changes? I have some async workarounds locally, so we could try them there too.

Yes, most of them seem like this. The ROOT CI is generally green.

[SahilPatidar]

Ok, I first need to understand this error. Do you know where this macro expands?

In file included from input_line_23:1:
/github/home/ROOT-CI/src/roottest/root/tree/selectorreader/runClasses.C:2:1: fatal error: 'SampleClasses_h' file not found
R__LOAD_LIBRARY(SampleClasses_h)
^
/github/home/ROOT-CI/build/include/Rtypes.h:458:35: note: expanded from macro 'R__LOAD_LIBRARY'
# define R__LOAD_LIBRARY(LIBRARY) _R_PragmaStr(cling load ( #LIBRARY ))
                                  ^
/github/home/ROOT-CI/build/include/Rtypes.h:457:26: note: expanded from macro '_R_PragmaStr'
# define _R_PragmaStr(x) _Pragma(#x)
                         ^
<scratch space>:4:33: note: expanded from here
 cling load ( "SampleClasses_h" )

[vgvassilev]

Ok, I first need to understand this error. Do you know where this macro expands?

In file included from input_line_23:1:
/github/home/ROOT-CI/src/roottest/root/tree/selectorreader/runClasses.C:2:1: fatal error: 'SampleClasses_h' file not found
R__LOAD_LIBRARY(SampleClasses_h)
^
/github/home/ROOT-CI/build/include/Rtypes.h:458:35: note: expanded from macro 'R__LOAD_LIBRARY'
# define R__LOAD_LIBRARY(LIBRARY) _R_PragmaStr(cling load ( #LIBRARY ))
                                  ^
/github/home/ROOT-CI/build/include/Rtypes.h:457:26: note: expanded from macro '_R_PragmaStr'
# define _R_PragmaStr(x) _Pragma(#x)
                         ^
<scratch space>:4:33: note: expanded from here
 cling load ( "SampleClasses_h" )

You can build the master and break at dlopen. The debugger will show you when and where it gets expanded.

[SahilPatidar]

I tried to reproduce the CI failure for the test roottest-root-aclic-misc-assertmyfun on Ubuntu 25.4. In CI it shows:

Error loading object: Incompatible object file: /github/home/ROOT-CI/build/roottest/root/aclic/misc/addIncludePathGCCMajor_C.so

This happens when the code tries to read symbols from a library that isn’t loaded yet (similar to ROOT/Cling’s Dyld::containSymbol). Normally we check if the library is shared or valid before registering it, so this error shouldn’t appear.

The same type of error is also showing up on other platforms:

• mac13 — test roottest-root-dataframe-test_reduce:
  Error loading object: No such file or directory
• alma9 arm64 — test roottest-root-tree-cache-CacheRange:
  Error loading object: The file was not recognized as a valid object file

These errors cause test failures due to error diffs. The Error loading object: message should only appear if something goes wrong in the middle of working with the shared library. This kind of error might not appear in Cling’s dyld implementation, since dyld doesn’t produce such errors.

To test locally, I ran on my Mac M1 with the same Docker image and steps used in CI:

docker run --security-opt label=disable --platform linux/amd64 -it registry.cern.ch/root-ci/ubuntu2504:buildready

Following the GitHub Actions steps, the test passed locally, so I couldn’t reproduce the failure outside CI.

[vgvassilev]

I tried to reproduce the CI failure for the test roottest-root-aclic-misc-assertmyfun on Ubuntu 25.4. In CI it shows:

Error loading object: Incompatible object file: /github/home/ROOT-CI/build/roottest/root/aclic/misc/addIncludePathGCCMajor_C.so

This happens when the code tries to read symbols from a library that isn’t loaded yet (similar to ROOT/Cling’s Dyld::containSymbol). Normally we check if the library is shared or valid before registering it, so this error shouldn’t appear.

The same type of error is also showing up on other platforms:

* **mac13** — test `roottest-root-dataframe-test_reduce`:
  `Error loading object: No such file or directory`

* **alma9 arm64** — test `roottest-root-tree-cache-CacheRange`:
  `Error loading object: The file was not recognized as a valid object file`

These errors cause test failures due to error diffs. The Error loading object: message should only appear if something goes wrong in the middle of working with the shared library. This kind of error might not appear in Cling’s dyld implementation, since dyld doesn’t produce such errors.

To test locally, I ran on my Mac M1 with the same Docker image and steps used in CI:

docker run --security-opt label=disable --platform linux/amd64 -it registry.cern.ch/root-ci/ubuntu2504:buildready

Following the GitHub Actions steps, the test passed locally, so I couldn’t reproduce the failure outside CI.

@dpiparo, we need help here with the reproduction logic I guess...

[aaronj0]

Hi @SahilPatidar looks like we have a relatively stable build. I'd like to point out that the two failing tests on macbeta:

roottest-root-treeformula-stl-writemap
roottest-root-treeformula-stl-mapvector

are unrelated to this PR. The TMVA test failures only on alma platforms seems strange, can you rebase on master?

[SahilPatidar]

Ok, I’ll do that. But I recently rebased it — if the alma tests are failing due to this PR, could you let me know what might be going wrong based on the test failure details?

[aaronj0]

Ok, I’ll do that. But I recently rebased it — if the alma tests are failing due to this PR, could you let me know what might be going wrong based on the test failure details?

The 3 failing tests belong to ROOT's machine learning component - TMVA. Two of the TMVA tests (pyunittests-bindings-pyroot-pythonizations-pyroot-pyz-sofie-gnn, tutorial-machine_learning-TMVA_SOFIE_GNN-py) show failed calls to cuInit on alma10 :

07:45:15.852599: E external/local_xla/xla/stream_executor/cuda/cuda_platform.cc:51] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)

And on both alma8 and alma10 we see a seg fault coming from libopenblas on tutorial-machine_learning-TMVA_SOFIE_GNN_Application which seems to be a remanifestation of some regex related issue...

Perhaps you can try reproducing these with the relevant docker images?

[ferdymercury]

if the alma tests are failing due to this PR, could you let me know what might be going wrong based on the test failure details?

I think they are failing in many other PR, too

[vgvassilev]

If that's so, we should move forward...

[aaronj0]

I think they are failing in many other PR, too

Yes, looks like none of the test failures here are related.

If that's so, we should move forward...

Yes, @SahilPatidar @vgvassilev I'd like to know how we proceed in this case. If the PR ready to land in LLVM, can we go ahead there? I assume we rework the DLM in Cling to use the new LibraryResolver with the next upgrade?

[vgvassilev]

I’d wait until landing this in llvm. I think we are quite close now.

[SahilPatidar]

One more task I need to complete is moving LLVM PR code from Orc/TargetProcess to Orc/Shared, so that it can be used on the Root side as an API.

[vgvassilev]

@SahilPatidar, can we also drop the relevant code that this PR is replacing?

[SahilPatidar]

Okay, I’m also going to push the concurrency-related issue fixes we encountered. Also in llvm side.