Switched to static linking of llvm by timostrunk · Pull Request #100 · conda-forge/llvmlite-feedstock

timostrunk · 2025-02-13T08:59:10Z

I switched to static linking of libllvm here and added it to the ignore_run_exports on unix. I did not change the build behaviour on Windows as it seems to already build with default settings there and I have no means to test it.

This fixes #99 and #84.

Reasons against this PR:

Obviously this will increase binary size, because libllvm is now statically linked. Im explicitly tagging @xhochy here, because he was against static linking this.
If the symbol version issue would be identified and resolved it would be cleaner

Reasons to merge this PR:

It resolves a hard to debug issue, which occurs using libraries, where debugging skill is required to find out, why a segmentation fault actually happened
It can easily be reverted, once the symbol isolation is gone.

Checklist

Used a personal fork of the feedstock to propose changes
Bumped the build number (if the version is unchanged)
Reset the build number to 0 (if the version changed)
Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
Ensured the license file is being packaged.

timostrunk · 2025-02-13T08:59:23Z

@conda-forge-admin, please rerender

conda-forge-admin · 2025-02-13T09:00:31Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.

timostrunk · 2025-02-13T09:32:37Z

ppc64le still broken, will look into it

timostrunk · 2025-02-13T10:46:57Z

ppc64le still broken, will look into it

build works locally on cross-compile and goes further than action runner. Test fail (as expected) due to invalid binary format.

timostrunk · 2025-02-14T11:11:18Z

I will try to dynamic link on ppc64le still. I think the native build has issues. The local crossbuild on my machine works though, but I don't want to sacrifice the tests, which I would need to do, if I let the github runner build on linux64.

Reverted the PR to draft, because there will be some noise.

timostrunk · 2025-02-14T12:30:24Z

@conda-forge-admin, please rerender

conda-forge-admin · 2025-02-14T12:32:46Z

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

_{This message was generated by GitHub Actions workflow run https://github.com/conda-forge/conda-forge-webservices/actions/runs/13329229741. Examine the logs at this URL for more detail.}

…nda-forge-pinning 2025.02.12.20.08.11

timostrunk · 2025-02-14T13:27:28Z

@jakirkham : This is now ready for review. ppc64le still builds shared, because I was unable to static link on a native ppc64le platform.

jakirkham

As already discussed at length previously ( #73 ), will not be accepting a change to static linking

Think we need to find a better way to address the issue users are encountering. One avenue may be changing the linker flags (stripping, symbol visibility, etc.)

timostrunk · 2025-02-19T22:04:05Z

My analysis showed that the cause of the issue might not be in llvmlite, so changing the flags here won't change much. The issue is that libllvm15 links into libllvm19, so I presume the right place for a fix is there. If my analysis is right libllvm15 should be in conflict with libllvm19, as it is not safe to have both in the same environment.

Until this fix is found it is desastrous for the conda-forge ecosystem not to provide a workaround. Currently not merging this means that people using pytorch and numba in the same conda environment either

face segmentation faults
need to downpin pytorch and keep the pin until llvmlite gets libllvm19 compatibility at which point this starts working again 'by chance' and will most probably break again with the next libllvm release.

numba and pytorch is a very common combination. Therefore: Please reconsider merging this. It can be reverted once the issue is solved.

timostrunk · 2025-02-19T22:13:46Z

Also in #73 you write:
"Users have been working with Numba on CUDA for some time without issues and this uses the dynamic linking solution we currently have

Not following the argument for static linking"

The reason this is working is only because numba and pytorch-cuda required the same libllvm version at this precise time. When libllvm14 was used by llvmlite and libllvm15 was used by pytorch-cuda, it broke. Now the same thing happens with libllvm15 and 19.

h-vetinari · 2025-02-19T22:14:26Z

As already discussed at length previously ( #73 ), will not be accepting a change to static linking

Pity you did not respond on the thread I opened in the core channel. I will bring this to a vote within core.

jakirkham · 2025-02-19T22:14:36Z

If this is indeed an LLVM issue, let's file on that feedstock. If there was already an issue filed and it was closed prematurely, am happy to reopen (provided a link)

timostrunk · 2025-02-19T22:16:38Z

This is the issue I opened on the llvm-dev feedstock: conda-forge/llvmdev-feedstock#312

beckermr · 2025-02-21T18:09:36Z

I am happy to take on the maintenance burden of pushing updates to the llvm versions for static linking if that helps unblock this.

beckermr · 2025-02-25T23:52:11Z

OK friends. I have another idea on how to ease the maintenance burden of static linking so we can unblock this PR.

I have added a new feature to the bot in this PR: conda-forge/conda-forge-bot#3755.

It allows the bot to update static libs according to an abstract spec as follows.

In the extra section you add this bit of yaml

extra:
  static_linking_host_requirements:
    - llvmdev 15.*
    - llvm 15.*

This specifies the abstract requirements for the static library you want in host.

Then in your recipe, you list the exact packages you care about like this

requirements:
  host:
    - python
    - setuptools
    - llvmdev 15.0.7 h2621b3d_4  # [osx and arm64]
    - llvm 15.0.7 h4a7a88c_4     # [osx and arm64]
    - llvmdev 15.0.7 hbedff68_4  # [osx and x86]
    - llvm 15.0.7 hed0f868_4     # [osx and x86]

The bot then does the following computation:

For each abstract static lib host requirement in the extra section

Get the latest version for all platforms the feedstock builds
If the latest versions+build numbers across all platforms differ, bail on whole update.

Extract the specs of packages in host in the current recipe that

match the abstract host requirements in the recipe per platform
are exact specs

If any versions and/or build numbers at equal version have increased when comparing 1 and 2

update the static libs in the recipe
issue a PR

The result of this is an update to the host section like this:

requirements:
  host:
    - python
    - setuptools
    - llvmdev 15.0.7 h4429f82_5  # [osx and arm64]
    - llvm 15.0.7 h0cf516b_5     # [osx and arm64]
    - llvmdev 15.0.7 hc29ff6c_5  # [osx and x86]
    - llvm 15.0.7 hb21d583_5     # [osx and x86]
    - zlib
    - vs2015_runtime  # [win]

The bot stores the new static lib versions it used to update the feedstock as part of the PR info it has. This should prevent it from issuing duplicate PRs.

Further, by bailing if not all of the versions+build numbers of the new static libs match, we should prevent PRs being issued in the middle of a build of the static libs on the backend.

Finally, by restricting the search for updates to the static libs to the abstract specs in extra, the bot will only issue an update for increases in minor+patch versions and/or build numbers in the say 15.* series for llvm, etc.

If this is of interest to you all, let me know and I can finish up the bot PR, make a PR to this feedstock, and we can start trying it out.

cc @isuruf @h-vetinari @jakirkham @conda-forge/llvmlite

h-vetinari · 2025-02-26T01:07:42Z

Thank you for trying to find a solution on this @beckermr! I'm fine with whatever setup that lets us get rid of the segfaults. To me the static_linking_host_requirements: machinery sounds a bit like overkill, but if it addresses the maintenance concerns here, then I'm happy to go in that direction.

conda-forge-admin · 2025-02-27T14:16:16Z

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found some lint.

Here's what I've got...

For recipe/meta.yaml:

❌ In conda-forge.yml: $.bot = {'update_static_libs': True, 'abi_migration_branches': ['rc']}.

{'update_static_libs': True, 'abi_migration_branches': ['rc']} is not valid under any of the given schemas
Schema
```
{
  "anyOf": [
    {
      "$ref": "#/$defs/BotConfig"
    },
    {
      "type": "null"
    }
  ]
}
```

_{This message was generated by GitHub Actions workflow run https://github.com/conda-forge/conda-forge-webservices/actions/runs/13569596802. Examine the logs at this URL for more detail.}

beckermr · 2025-02-27T14:32:19Z

@isuruf ppc64le is not statically linked in these builds due to some error @timostrunk had when they tried above.

beckermr · 2025-02-27T15:01:29Z

PR to fix the linter: conda-forge/conda-smithy#2253

beckermr · 2025-02-27T15:29:16Z

OK. This one is all green or will be soon. I do not want to merge when another maintainer has requested changes.

@jakirkham Can you look at what we've done here and reconsider your review possibly? We've enabled fully automatic updates via the bot and I've added myself to the feedstock to manage things around that as I expect we'll encounter a few bugs along the way.

beckermr · 2025-03-06T15:02:46Z

@conda-forge-admin relint

conda-forge-admin · 2025-03-06T15:04:31Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe/meta.yaml) and found it was in an excellent condition.

timostrunk · 2025-03-11T14:17:00Z

Pinging @jakirkham again here to get the discussion rolling again. Could you please comment on whether you are ok with this setup?

kkraus14 · 2025-03-12T17:02:10Z

+elif [[ "${target_platform}" == linux-ppc64le ]]; then
+    CXXFLAGS="${CXXFLAGS} -fplt"


@isuruf Any idea why this was needed? Would be good to leave a comment.

Think this is because we normally pass -fno-plt in the CXXFLAGS. This actually done for all Linux architectures

However there have been some cases where this doesn't work on linux_ppc64le (notably with LLVM). Please see this bug:

LLVM fails to build on PPC with GCC>=9 and -fno-plt llvm/llvm-project#51205

Think Isuru is adding -fplt as a quick way of overriding the -fno-plt behavior. Adding -fplt is a bit quicker than what we normally do, which is remove the -fno-plt flag

Since we have figured out that this works, think we should adopt the syntax that we have elsewhere and remove -fno-plt from flags. This will also make it easier for future readers to find more context

Suggested change

elif [[ "${target_platform}" == linux-ppc64le ]]; then

CXXFLAGS="${CXXFLAGS} -fplt"

elif [[ "${target_platform}" == linux-ppc64le ]]; then

# Taken from llvmdev's recipe

# https://github.com/conda-forge/llvmdev-feedstock/blob/8c2c0f2db9db1fdf12289381dcee4e2d9a2e5fec/recipe/build.sh#L29-L33

# disable `-fno-plt` due to some GCC bug causing linker errors, see

# https://github.com/llvm/llvm-project/issues/51205

CFLAGS="$(echo $CFLAGS | sed 's/-fno-plt //g')"

CXXFLAGS="$(echo $CXXFLAGS | sed 's/-fno-plt //g')"

Should add have committed the syntax change above. Though leaving unresolved so the thread remains visible

kkraus14 · 2025-03-12T17:09:10Z

Had a side chat related to this PR as well as consulted @gmarkall who is a numba and llvmlite maintainer:

If a solution to dynamic linking properly and safely is found, we should move back to dynamic linking as it is generally preferred and makes some things easier.
In addition to just static linking, you need to hide the visibility of LLVM symbols, otherwise you can run into issues like static symbols alias despite symbol versioning across two libLLVM's llvm/llvm-project#47565. llvmlite already hides the symbols: https://github.com/numba/llvmlite/blob/2677283bf7500916606e7663611ce7f076d7be93/ffi/CMakeLists.txt#L72
I'm still not 100% on why this is happening as the libLLVM-15 and libLLVM-19 symbols are both versioned with version specific identifiers. A backtrace of the reproducer shows it properly hitting the function from libLLVM-15 and that deep in the call stack of LLVM it starts hitting functions from libLLVM-19 instead. I am not clear on why this is happening.

gmarkall · 2025-03-13T12:04:51Z

I'm still not 100% on why this is happening as the libLLVM-15 and libLLVM-19 symbols are both versioned with version specific identifiers.

I think only one version of the symbol can exist in a process - so, my understanding is that if a symbol has already been resolved with the LLVM 15-specific version, it's not going to be resolved again if an LLVM 19-specific caller has a relocation to a symbol of the same name that subsequently needs to be resolved. So symbol versioning doesn't help in our situation here.

beckermr · 2025-03-14T20:33:34Z

After some sidebar conversations with various folks, I think we've reached a consensus of sorts.

The plan as I understand it is to:

Merge this PR sometime in the next 1-2 weeks. Waiting on approval from @jakirkham IIUIC.
Various folks working on numba, conda-forge, etc. will continue to investigate the LLVM issue.
@jakirkham suggested to me offline that we can merge this PR and then push a test build number bump to llvm in order to test out the bot integrations. I am happy to baby sit the bot on that and iron our bugs if people have interest. This will need to wait until the week after next week, but that test should not block this PR.

Thanks everyone for working hard on this tricky issue!

beckermr · 2025-03-21T13:11:38Z

Per further discussion, I will merge this PR on Monday, March 24, 2025. Happy weekend!

jakirkham

Apologies for the slow reply here

Met late last week with both Keith and Matt to discuss the changes and maintenance here

Agree that static linking is the least bad option we can come up with atm. So agree we should do that

There were a couple questions that had come up when we looked at some of the changes here. It took a bit longer to dig into these. Have made comments on them below

Also agree it would be great to have Matt on as a maintainer. Would like to add Keith as well (if he agrees). This should help with keeping up on changes here

jakirkham · 2025-03-14T18:34:36Z

+elif [[ "${target_platform}" == linux-ppc64le ]]; then
+    CXXFLAGS="${CXXFLAGS} -fplt"


Think this is because we normally pass -fno-plt in the CXXFLAGS. This actually done for all Linux architectures

However there have been some cases where this doesn't work on linux_ppc64le (notably with LLVM). Please see this bug:

LLVM fails to build on PPC with GCC>=9 and -fno-plt llvm/llvm-project#51205

Think Isuru is adding -fplt as a quick way of overriding the -fno-plt behavior. Adding -fplt is a bit quicker than what we normally do, which is remove the -fno-plt flag

Since we have figured out that this works, think we should adopt the syntax that we have elsewhere and remove -fno-plt from flags. This will also make it easier for future readers to find more context

Suggested change

elif [[ "${target_platform}" == linux-ppc64le ]]; then

CXXFLAGS="${CXXFLAGS} -fplt"

elif [[ "${target_platform}" == linux-ppc64le ]]; then

# Taken from llvmdev's recipe

# https://github.com/conda-forge/llvmdev-feedstock/blob/8c2c0f2db9db1fdf12289381dcee4e2d9a2e5fec/recipe/build.sh#L29-L33

# disable `-fno-plt` due to some GCC bug causing linker errors, see

# https://github.com/llvm/llvm-project/issues/51205

CFLAGS="$(echo $CFLAGS | sed 's/-fno-plt //g')"

CXXFLAGS="$(echo $CXXFLAGS | sed 's/-fno-plt //g')"

jakirkham · 2025-03-22T02:16:50Z

    - marcelotrevisani
    - xhochy
    - mbargull
+    - beckermr


@kkraus14 is it ok if we add you to the maintainers here?

Suggested change

- beckermr

- beckermr

- kkraus14

jakirkham · 2025-03-22T02:23:01Z

@conda-forge-admin , please re-render

…nda-forge-pinning 2025.03.21.21.56.39

(submitted an updated review)

beckermr · 2025-03-24T11:22:26Z

@conda-forge-admin rerender

conda-forge-admin · 2025-03-24T11:24:47Z

Hi! This is the friendly automated conda-forge-webservice.

I tried to rerender for you, but it looks like there was nothing to do.

_{This message was generated by GitHub Actions workflow run https://github.com/conda-forge/conda-forge-webservices/actions/runs/14034002527. Examine the logs at this URL for more detail.}

beckermr · 2025-03-24T11:25:42Z

I am happy to add @kkraus14 in another PR if he'd like. Going to merge.

timostrunk requested review from jakirkham, marcelotrevisani, mbargull, souravsingh and xhochy as code owners February 13, 2025 08:59

timostrunk mentioned this pull request Feb 13, 2025

Linking libllvm-19 before using libllvmlite (currently requiring libllvm-15) leads to segmentation faults. #99

Closed

1 task

timostrunk marked this pull request as draft February 14, 2025 10:57

timostrunk force-pushed the switch_to_static branch from 05f65d3 to 7673ab6 Compare February 14, 2025 12:37

timostrunk and others added 2 commits February 14, 2025 13:56

Switched to static linking of llvm for linux-64 and osx-*

67353fe

MNT: Re-rendered with conda-build 25.1.2, conda-smithy 3.45.4, and co…

2a97a9a

…nda-forge-pinning 2025.02.12.20.08.11

timostrunk force-pushed the switch_to_static branch from 7673ab6 to 2a97a9a Compare February 14, 2025 13:08

timostrunk marked this pull request as ready for review February 14, 2025 13:09

timostrunk mentioned this pull request Feb 14, 2025

Unnecessary libllvm dependency for numba through llvmlite #84

Closed

1 task

jakirkham previously requested changes Feb 19, 2025

View reviewed changes

Update build.sh

af6574a

Update build.sh

03326d4

beckermr mentioned this pull request Feb 27, 2025

static linkage migrations conda-forge/conda-forge-bot#3747

Closed

kkraus14 approved these changes Mar 12, 2025

View reviewed changes

h-vetinari mentioned this pull request Mar 15, 2025

update to 2.6.0 conda-forge/torchaudio-feedstock#16

Closed

5 tasks

Hneuschmidt mentioned this pull request Mar 17, 2025

Segmentation Fault on applications that use PyQt5 and the xcube plugin system xcube-dev/xcube#1138

Open

gmarkall mentioned this pull request Mar 18, 2025

numba and pytorch: LLVM dynamic loading crash numba/numba#9996

Closed

2 tasks

jakirkham reviewed Mar 22, 2025

View reviewed changes

Match common -fno-plt flag removal on PPC

c6f123a

MNT: Re-rendered with conda-build 25.1.2, conda-smithy 3.47.0, and co…

60ed7d3

…nda-forge-pinning 2025.03.21.21.56.39

jakirkham reviewed Mar 22, 2025

View reviewed changes

Comment thread recipe/meta.yaml

beckermr merged commit 85bc9a2 into conda-forge:main Mar 24, 2025

esc mentioned this pull request Sep 1, 2025

openai-whisper: use llvm@20 (pre-release testing) Homebrew/homebrew-core#235317

Closed

2 tasks

gmarkall mentioned this pull request Oct 21, 2025

SegmentationFault with llvmlite 0.45.1 + qgis 3.44.3 #109

Closed

1 task

		elif [[ "${target_platform}" == linux-ppc64le ]]; then
		CXXFLAGS="${CXXFLAGS} -fplt"

-elif [[ "${target_platform}" == linux-ppc64le ]]; then
-    CXXFLAGS="${CXXFLAGS} -fplt"
+elif [[ "${target_platform}" == linux-ppc64le ]]; then
+    # Taken from llvmdev's recipe
+    # https://github.com/conda-forge/llvmdev-feedstock/blob/8c2c0f2db9db1fdf12289381dcee4e2d9a2e5fec/recipe/build.sh#L29-L33
+    # disable `-fno-plt` due to some GCC bug causing linker errors, see
+    # https://github.com/llvm/llvm-project/issues/51205
+    CFLAGS="$(echo $CFLAGS | sed 's/-fno-plt //g')"
+    CXXFLAGS="$(echo $CXXFLAGS | sed 's/-fno-plt //g')"

Uh oh!

Uh oh!

Conversation

timostrunk commented Feb 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

timostrunk commented Feb 13, 2025

Uh oh!

conda-forge-admin commented Feb 13, 2025

Uh oh!

timostrunk commented Feb 13, 2025

Uh oh!

timostrunk commented Feb 13, 2025

Uh oh!

timostrunk commented Feb 14, 2025

Uh oh!

timostrunk commented Feb 14, 2025

Uh oh!

conda-forge-admin commented Feb 14, 2025

Uh oh!

timostrunk commented Feb 14, 2025

Uh oh!

jakirkham left a comment

Choose a reason for hiding this comment

Uh oh!

timostrunk commented Feb 19, 2025

Uh oh!

timostrunk commented Feb 19, 2025

Uh oh!

h-vetinari commented Feb 19, 2025

Uh oh!

jakirkham commented Feb 19, 2025

Uh oh!

timostrunk commented Feb 19, 2025

Uh oh!

beckermr commented Feb 21, 2025

Uh oh!

beckermr commented Feb 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

h-vetinari commented Feb 26, 2025

Uh oh!

conda-forge-admin commented Feb 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

beckermr commented Feb 27, 2025

Uh oh!

beckermr commented Feb 27, 2025

Uh oh!

beckermr commented Feb 27, 2025

Uh oh!

beckermr commented Mar 6, 2025

Uh oh!

conda-forge-admin commented Mar 6, 2025

Uh oh!

timostrunk commented Mar 11, 2025

Uh oh!

kkraus14 Mar 12, 2025

Choose a reason for hiding this comment

Uh oh!

jakirkham Mar 14, 2025

Choose a reason for hiding this comment

Uh oh!

jakirkham Mar 22, 2025

Choose a reason for hiding this comment

Uh oh!

kkraus14 commented Mar 12, 2025

Uh oh!

gmarkall commented Mar 13, 2025

Uh oh!

beckermr commented Mar 14, 2025

Uh oh!

beckermr commented Mar 21, 2025

Uh oh!

jakirkham left a comment

Choose a reason for hiding this comment

Uh oh!

jakirkham Mar 14, 2025

Choose a reason for hiding this comment

Uh oh!

timostrunk commented Feb 13, 2025 •

edited

Loading

beckermr commented Feb 25, 2025 •

edited

Loading

conda-forge-admin commented Feb 27, 2025 •

edited

Loading