Solve the pinned memchr problem

[ehuss]

Ever since #120454, the memchr crate has been pinned at 2.5.0. This is causing problems when trying to update things which depend on a newer version (like regex). This issue is to track trying to get that problem resolved.

The dependency is pinned because the x86_64-pc-windows-gnu target fails some tests with the 2.6.0 release. These tests fail when linking with "undefined reference" symbol errors in memchr (which ends up in the standard library via the object dependency for backtraces).

The errors look like:

...ld.exe: ...-cgu.0.rcgu.o.rcgu.o:std.33d504e26fbb75:(.text+0x66963): undefined reference to `__imp__ZN6memchr4arch6x86_646memchr10memchr_raw2FN17h0aa7d2dff3bebec3E'

The underlying problem isn't with memchr. It's just that version 2.6 is just triggering a bug that has been around for a long while due to some refactoring of the code.

Reproduction

1. Edit rustc_ast/Cargo.toml and remove the = on memchr.

2. cargo update -p memchr

3. ./x test tests/ui --build=x86_64-pc-windows-msvc --host=x86_64-pc-windows-gnu

   • AFAIK, the defaults in config.toml should be fine (no profile).
   • As a reminder, to use the same mingw from CI, download https://ci-mirrors.rust-lang.org/rustc/x86_64-12.2.0-release-posix-seh-rt_v10-rev0.7z, extract that and add the bin directory to your PATH.

This currently fails these tests:

    [ui] tests\ui\extern\issue-64655-extern-rust-must-allow-unwind.rs#thin0
    [ui] tests\ui\extern\issue-64655-extern-rust-must-allow-unwind.rs#thin1
    [ui] tests\ui\extern\issue-64655-allow-unwind-when-calling-panic-directly.rs#thin
    [ui] tests\ui\extern\issue-64655-extern-rust-must-allow-unwind.rs#thin2
    [ui] tests\ui\extern\issue-64655-extern-rust-must-allow-unwind.rs#thin3
    [ui] tests\ui\lto\all-crates.rs
    [ui] tests\ui\lto\lto-thin-rustc-loads-linker-plugin.rs
    [ui] tests\ui\lto\thin-lto-inlines2.rs

Theories

This is likely some interaction with Thin LTO and inline.

A theory is that this is a bug in ld.bfd (just based on the evidence that it works with lld). However, it may also be possible that rustc is generating things in an incorrect way, and lld is just more tolerant of it.

Note that memchr is doing some fancy things with function pointers (https://github.com/BurntSushi/memchr/blob/2.7.4/src/arch/x86_64/memchr.rs#L58-L160) with inline(always) functions.

Notes

• As far as we know, updating binutils does not fix it. At the time of this writing, we are using version 2.38.
• https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/.60ui_test.60.20v0.2E22.2E1.20version.20bump.20causes.20dep.20conflict/ contains an early discussion of this problem.
• https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/Stuck.20memchr.202.2E5.2E0.20saga Another discussion that lead to this issue.
• windows-gnu thin-lto test linking error regression #109797 -- windows-gnu thin-lto test linking error regression
  • I'm not sure if this is exactly the same issue, but seems very similar.
  • The thread-local inlining issue has a workaround in Workaround #109797 on windows-gnu #109806
• linking issue on x86_64-pc-windows-gnu target + thinLTO related to static function pointers #98302 -- linking issue on x86_64-pc-windows-gnu target + thinLTO related to static function pointers
  • Unsure if this is the exact same issue.
• cargo update #120454 -- Pinned memchr to 2.5.0
• The unrecognized directive and corrupt .drectve warnings can (probably?) be ignored, they are only due to using an older version of binutils (GNU linker warns “corrupt .drectve” on staticlib binary generated with Rust 1.70 on Windows #112368).

Possible solutions

• Demote *-pc-windows-gnu and replace with *-pc-windows-gnullvm. To the best of our understanding, lld does not have this problem.
• Report this to the binutils maintainers to see if it is possible to fix it in ld.bfd.
• Is it possible to switch to gold or lld?
• Disable the tests on *-pc-windows-gnu until someone can fix the problem. This means any user using thin-lto will likely run into the same problem. Perhaps this will motivate one of them to try to fix it? 😉

Non-solutions

• We could change the inline attribute in memchr for windows-gnu, but nobody wants to do that. It's not memchr's fault, and we don't want to paper over the real problem.