Several upstream bottles cause segfault. related to old patchelf.rb bugs.
#163826
Comments
|
This issue seems related: #136432 |
|
I think the root cause related to many homebrew Linux segmentation faults is the homebrew build bot. The build bot making the bottles is still somehow using an older |
|
Latest patchelf updates in brew: Most issues have been fixed with version 1.4.0 in Nov. 2022. Why do you think it's still using an old patchelf.rb version? In your config:
This looks old. Can you update first? |
It's not an issue with See above the part with
How do I do that? I ran |
|
|
Don't worry about the 3 months old update message: you are now using the API to get the latest version, so I guess no need to update the git repo on your computer to get the latest version. I think that message is confusing, I'll have a look into it later. I think I know what happened.
We will have to rebuild these bottles. I have triggered a Once this is done, you can |
|
Sounds great. There are several other bottles with crashes, such as golang. |
|
👋 Been quietly watching this and previous issues. In WSL Ubuntu this is pretty wide spread from packages built with go and those that are built with gcc. I believe I saw less issues when I tested in a Ubuntu VM. What could be an effective way forward for the project maintainers? I imagine as the underlying packages upgrade, they'll be rebuilt, so as users, should we just wait? Would the repo owners find a list of affected packages be helpful? I'd be happy to send along the packages I've had issues with and help UAT rebuilds. |
|
@redspot a new
Yes, if you have a list, we can at least rebuild all of these. Most packages get updates / rebuilds at least once per year, but some others will take time to get update. Let's fix what is known to be broken. |
|
Here's a list from my bash/zsh history, though I think I had an issue with every package that uses
|
|
see this comment for more potentially broken bottles #132852 (comment) I definitely had issues with these formulas:
but, there could have been others. It's not that I would like you to fix a specific formula. It's that you most likely have many broken bottles that need a rebuild. I guess you could come up with a way to find bottles that are:
|
|
Can just someone confirm that Note: we probably won't rebottle everything, I'll go through the lists above, for anything else users will have to open an issue or post here. |
|
Sure, I'll test out the new bottle on Monday. I could always solve the problem building from source. This ticket was just to give the project and other users a heads up. |
|
@iMichka I installed Output$ brew install ssdeep
...
==> Downloading https://ghcr.io/v2/homebrew/core/ssdeep/manifests/2.14.1-1
#################################################################################################################################################################################################################################### 100.0%
==> Fetching ssdeep
==> Downloading https://ghcr.io/v2/homebrew/core/ssdeep/blobs/sha256:d62ce2005fd901e4a363b3f6592dee0388e9114d323a058499d16e8dba9c4d38
#################################################################################################################################################################################################################################### 100.0%
==> Pouring ssdeep--2.14.1.x86_64_linux.bottle.1.tar.gz
� /home/linuxbrew/.linuxbrew/Cellar/ssdeep/2.14.1: 16 files, 215.4KB
==> Running `brew cleanup ssdeep`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
$ /home/linuxbrew/.linuxbrew/bin/ssdeep -h
ssdeep version 2.14.1 by Jesse Kornblum and the ssdeep Project
For copyright information, see man page or README.TXT.
Usage: ssdeep [-m file] [-k file] [-dpgvrsblcxa] [-t val] [-h|-V] [FILES]
-m - Match FILES against known hashes in file
-k - Match signatures in FILES against signatures in file
-d - Directory mode, compare all files in a directory
-p - Pretty matching mode. Similar to -d but includes all matches
-g - Cluster matches together
-v - Verbose mode. Displays filename as its being processed
-r - Recursive mode
-s - Silent mode; all errors are suppressed
-b - Uses only the bare name of files; all path information omitted
-l - Uses relative paths for filenames
-c - Prints output in CSV format
-x - Compare FILES as signature files
-a - Display all matches, regardless of score
-t - Only displays matches above the given threshold
-h - Display this help message
-V - Display version number and exit
$ /home/linuxbrew/.linuxbrew/bin/ssdeep -V
2.14.1 |
Yes, it works from the bottle. |
|
I've launched a couple of rebottles for formulas that were listed above and where the rebottling won't erase any old bottle (for previous macOS versions). In those cases, there is no downside to rebottling. Then, we can confirm whether the issues are all fixed, and decide how to proceed next. I've launched: |
|
❌ @fxcoudert bottle request for gh failed. |
|
❌ @fxcoudert bottle request for k9s failed. |
|
@fxcoudert I've tried a few of these ( $ k6 --version # previously built from source
k6 v0.49.0 (go1.21.6, linux/amd64)
$ brew update && brew reinstall k6
Already up-to-date.
==> Downloading https://ghcr.io/v2/homebrew/core/k6/manifests/0.49.0
####################################################################################################################################################################################################################### 100.0%
==> Fetching k6
==> Downloading https://ghcr.io/v2/homebrew/core/k6/blobs/sha256:2e548198d61b1f5336646ff43bf7f2cd75ed69c288920c5d47d18f3219c40e65
####################################################################################################################################################################################################################### 100.0%
==> Reinstalling k6
==> Pouring k6--0.49.0.x86_64_linux.bottle.tar.gz
==> Caveats
zsh completions have been installed to:
/home/linuxbrew/.linuxbrew/share/zsh/site-functions
==> Summary
� /home/linuxbrew/.linuxbrew/Cellar/k6/0.49.0: 8 files, 37.6MB
==> Running `brew cleanup k6`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
$ k6 --version
zsh: segmentation fault k6 --version
$ brew update && brew reinstall -s k6
Already up-to-date.
Warning: building from source is not supported!
You're on your own. Failures are expected so don't create any issues, please!
==> Fetching k6
==> Downloading https://raw.githubusercontent.com/Homebrew/homebrew-core/726ba7780161a49c1d6b4b202f854e13fb5dca66/Formula/k/k6.rb
####################################################################################################################################################################################################################### 100.0%
==> Downloading https://github.com/grafana/k6/archive/refs/tags/v0.49.0.tar.gz
Already downloaded: /home/pdelre/.cache/Homebrew/downloads/cb0f5adcdfa30d830e9f759aeb1266ce066200d6e60a225a1a5fb2bf7d8ca74c--k6-0.49.0.tar.gz
==> Reinstalling k6
==> go build -ldflags=-s -w
==> Caveats
zsh completions have been installed to:
/home/linuxbrew/.linuxbrew/share/zsh/site-functions
==> Summary
� /home/linuxbrew/.linuxbrew/Cellar/k6/0.49.0: 8 files, 38MB, built in 1 minute 8 seconds
==> Running `brew cleanup k6`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
$ k6 --version
k6 v0.49.0 (go1.22.0, linux/amd64) |
|
I don't know enough about how go executables are built & linked to be able to answer that. go is being rebuilt as part of this PR #165116 so we'll know soon. |
|
@pdelre perhaps you could |
|
@redspot Yes, that's how I resolve the builds locally. |
|
the root cause is with the CGO_ENABLED for go build, we should turn it all for all affected formulae. see https://github.com/grafana/k6/blob/master/release%20notes/v0.26.0.md?plain=1#L127 (for k6 for example) |
|
❌ @fxcoudert bottle request for go-task failed. |
|
@Bo98 That patch works on my WSL2 Ubuntu, Kernel 4. |
This works on my aarch64, CentOS 7, linux 4.19.91. |
|
@Bo98 Ok, I'll give that patch a try. First, I'll make sure everything is up-to-date: So, it seems like the patch is already applied upstream. Next, I'll try the steps suggested by @Shaloc : Let's check No segfault. Now, we try bottling: Also no segfault. And, finally let's check the virtual address maps from the ELF header: No segments match virtual address and, here's some kernel info: |
|
@Bo98 Thank you for your patch solution as it's continued to work through multiple upgrades (including It appears an upstream contribution must be made to https://github.com/david942j/patchelf.rb and that you were the author of the NixOS |
|
Yes, I will be getting things moving this weekend. There was a patch for a |
|
the |
|
I just migrated a Centos 8 Stream to Rocky Linux, and encountered the While debugging I started with the But I'm dropping this here in case it can help the debugging and/or reproducing efforts. docker run \ # ...or nerdctl
-ti \
--rm \
--platform amd64 \
-v $HOME/.docker/certs.d:/etc/pki/ca-trust/source/anchors \ # My env requires a trust CA, remove at will
rockylinux:8 bash -c '
update-ca-trust # My env requires a trust CA, remove at will
dnf -y install git procps sudo
dnf -y groupinstall "Development Tools"
echo "brewtest ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/brewtest
useradd -m brewtest
sudo su - brewtest -c /bin/bash -c "NONINTERACTIVE=1 $(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
(echo; echo "eval \"$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)\"") >> /home/brewtest/.bashrc
sudo su - brewtest -c /bin/bash -c "brew install gcc"
sudo su - brewtest -c /bin/bash -c "brew install gh"
sudo su - brewtest -c /bin/bash -c "gh"
'Now, in my server env the |
This comment was marked as outdated.
This comment was marked as outdated.
|
After recently upgrading recompiling |
|
|
It seems to effect the treesit support (e.g. rust-ts-mode) of emacs. |
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
|
Issues were fixed in Homebrew 4.2.20, with success reported from multiple users above. It was released fully (after a short period on the developer stream) on 29th of April and there's been little comments here since (treesit report I believe is a separate issue). I've tested it again under 4.3.7 with this kernel & OS: and was successful running: whereas it previously failed on 4.2.19. This seems to be an almost identical kernel to what you are running, so there must be something very specific to that RHEL 8.9 setup that even its sister OS CentOS 8 doesn't exhibit. |
|
I'll test it tomorrow. Thanks for the update |
|
Here is the latest info from Here is a version test for various programs: Everything seems to work, including So far, Compiling with Note the extra But, Issue not fully fixed yet. |
|
It would have been nice if the fixes in Homebrew 4.2.20 had been mentioned here as this is the issue thread that I'm subscribed to. The absence of any update gave the appearance that this issue was being ignored or would never get any fix. My own issue #136432 which was closed because of similarity to this has been resolved. 'go' package no longer segfaults. |
|
While I developed the initial fix, it required a few hops to merge (the patch initially had to go upstream first) and the eventual final release (4.2.20) was made when I was holiday, I apologise for this as I should have remembered to login and close this issue. It's good to know that the issue is fixed with exception to |
|
I really have no idea how to debug which memory mappings are overlapping. different kernels act differently. here's here's On the newer kernel, the fixed mappings are ignored, which is why there's no overlapping memory maps. this is the relevant kernel commit: torvalds/linux@a4ff8e8 this shows that the flag however, so, because of https://github.com/torvalds/linux/blob/a4ff8e8620d3f4f50ac4b41e8067b7d395056843/mm/mmap.c#L1376, the program crashes during ELF loading, meaning there's no way to trace what's getting mapped. The whole point of me investigating this is figure out how to make a test that can check to see if an ELF file would crash due to overlaps on kernel so, I don't know how to look for ELFs that would crash, other than running them, like with Sorry, I'm stumped. |
|
Also, can the |
ZhongRuoyu
added
help wanted
This comment was marked as off-topic.
This comment was marked as off-topic.
and others
brew gist-logs <formula>link ORbrew configANDbrew doctoroutputVerification
brew doctoroutput" saysYour system is ready to brew.and am still able to reproduce my issue.brew updateand am still able to reproduce my issue.brew doctorand that did not fix my problem.What were you trying to do (and why)?
I was attempting to use several formulae installed via
brew.There are several related issues:
#137991 related to old
patchelf.rbbug#132852 bottles crashing on Centos
#116841 bottles crashing on RHEL
What happened (include all command output)?
Note the segment that causes the problem,
0x0000000000401000Now, take a look at the on-disk segments:
Note the segment
0x00000000004012aeAlso, of interest, when run using
brew's interpreter, programs seem to work:Here is the results when built from source:
What did you expect to happen?
I expect the upstream pre-built bottles to not have lingering issues with
patchelf.rb.Now, the issues with
patchelf.rbhave been fixed, and maybe those Centos/RHEL related issues as well.It's that the bottles still have the problem.
Step-by-step reproduction instructions (by running
brewcommands)As mentioned in this comment: #132852 (comment)
Some formulas work:
Some formulas fail:
Note the kernel reporting the same segment,
0x0000000000401000, and readelf showing that segment. And, it strangely works when run withbrew's interpreter.when building from source:
It also seems reliable to detect the bad packages by looking for a segment that matches the regex
0x0000000000401....The text was updated successfully, but these errors were encountered: