feature `sha1/asm` is enabled on windows

[NobodyXu]

Duplicates

• I have searched the existing issues

Current behavior 😯

Compiling on windows failed failed with errors from sha1-asm, which is known in RustCrypto/asm-hashes#17 and should be migrated by gix-features by disabling sha1/asm on windows:

warning: cl : Command line warning D9024 : unrecognized source file type 'src/x64.S', object file assumed
warning: cl : Command line warning D9027 : source file 'src/x64.S' ignored

error: failed to run custom build command for `sha1-asm v0.5.1`

Caused by:
  process didn't exit successfully: `D:\a\cargo-binstall\cargo-binstall\target\release\build\sha1-asm-898e65df03af5add\build-script-build` (exit code: 1)
  --- stdout
  TARGET = Some("x86_64-pc-windows-msvc")
  OPT_LEVEL = Some("z")
  HOST = Some("x86_64-pc-windows-msvc")
  cargo:rerun-if-env-changed=CC_x86_64-pc-windows-msvc
  CC_x86_64-pc-windows-msvc = None
  cargo:rerun-if-env-changed=CC_x86_64_pc_windows_msvc
  CC_x86_64_pc_windows_msvc = None
  cargo:rerun-if-env-changed=HOST_CC
  HOST_CC = None
  cargo:rerun-if-env-changed=CC
  CC = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64-pc-windows-msvc
  CFLAGS_x86_64-pc-windows-msvc = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64_pc_windows_msvc
  CFLAGS_x86_64_pc_windows_msvc = None
  cargo:rerun-if-env-changed=HOST_CFLAGS
  HOST_CFLAGS = None
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = None
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  CARGO_CFG_TARGET_FEATURE = Some("fxsr,sse,sse2")
  DEBUG = Some("false")
  running: "C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise\\VC\\Tools\\MSVC\\14.36.32532\\bin\\HostX64\\x64\\cl.exe" "-nologo" "-MD" "-O1" "-Brepro" "-W4" "-c" "-FoD:\\a\\cargo-binstall\\cargo-binstall\\target\\x86_64-pc-windows-msvc\\release\\build\\sha1-asm-f3dce8890bea4dcb\\out\\src/x64.o" "-c" "src/x64.S"
  cargo:warning=cl : Command line warning D9024 : unrecognized source file type 'src/x64.S', object file assumed
  cargo:warning=cl : Command line warning D9027 : source file 'src/x64.S' ignored
  cl : Command line warning D9021 : no action performed
  exit code: 0
  cargo:rerun-if-env-changed=AR_x86_64-pc-windows-msvc
  AR_x86_64-pc-windows-msvc = None
  cargo:rerun-if-env-changed=AR_x86_64_pc_windows_msvc
  AR_x86_64_pc_windows_msvc = None
  cargo:rerun-if-env-changed=HOST_AR
  HOST_AR = None
  cargo:rerun-if-env-changed=AR
  AR = None
  cargo:rerun-if-env-changed=ARFLAGS_x86_64-pc-windows-msvc
  ARFLAGS_x86_64-pc-windows-msvc = None
  cargo:rerun-if-env-changed=ARFLAGS_x86_64_pc_windows_msvc
  ARFLAGS_x86_64_pc_windows_msvc = None
  cargo:rerun-if-env-changed=HOST_ARFLAGS
  HOST_ARFLAGS = None
  cargo:rerun-if-env-changed=ARFLAGS
  ARFLAGS = None
  running: "C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise\\VC\\Tools\\MSVC\\14.36.32532\\bin\\HostX64\\x64\\lib.exe" "-out:D:\\a\\cargo-binstall\\cargo-binstall\\target\\x86_64-pc-windows-msvc\\release\\build\\sha1-asm-f3dce8890bea4dcb\\out\\libsha1.a" "-nologo" "D:\\a\\cargo-binstall\\cargo-binstall\\target\\x86_64-pc-windows-msvc\\release\\build\\sha1-asm-f3dce8890bea4dcb\\out\\src/x64.o"
  LINK : fatal error LNK1181: cannot open input file 'D:\a\cargo-binstall\cargo-binstall\target\x86_64-pc-windows-msvc\release\build\sha1-asm-f3dce8890bea4dcb\out\src\x64.o'
  exit code: 1181

  --- stderr


  error occurred: Command "C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise\\VC\\Tools\\MSVC\\14.36.32532\\bin\\HostX64\\x64\\lib.exe" "-out:D:\\a\\cargo-binstall\\cargo-binstall\\target\\x86_64-pc-windows-msvc\\release\\build\\sha1-asm-f3dce8890bea4dcb\\out\\libsha1.a" "-nologo" "D:\\a\\cargo-binstall\\cargo-binstall\\target\\x86_64-pc-windows-msvc\\release\\build\\sha1-asm-f3dce8890bea4dcb\\out\\src/x64.o" with args "lib.exe" did not execute successfully (status code exit code: 1181).


warning: build failed, waiting for other jobs to finish...
      Timing report saved to D:\a\cargo-binstall\cargo-binstall\target\cargo-timings\cargo-timing-20230629T142057Z.html

Expected behavior 🤔

Build should succeed.

Steps to reproduce 🕹

Compiling gix on x64/arm windows with features max-performance enabled.

[Byron]

Thanks a lot for letting me know! Is this something you would feel comfortable to contribute? It seems like this project's CI is missing that environment or configuration. I will also take a stab at it and at least try to get a reproduction on this project's CI, from there fixing it should get easier.

[NobodyXu]

Yes, I will have a try once I have time.
Reproducing it in CI should be easy.

[NobodyXu]

To be honest, I think this might be a upstream bug.
I googled it and this discussion from 2015 says target_env is internal to compiler, but it's clearly documented in user facing document.

Given that gix-features will not enable sha1/asm unless not(target_env = "msvc") and on x86/arm, it could only be an upstream bug.

[NobodyXu]

I think we can workaround this by doing not(target = "x86_64-windows-msvc"), not(target = "aarch64-windows-msvc") instead of not(target_env = "msvc") given that sha1/asm only supports arm 64 and x86 anyway.

But that will require quite a list of targets since there are many x86 targets (i686, i484)

[Byron]

What I don't understand about this is that as per the gix-features/Cargo.toml, it already excludes the respective environment:

[target.'cfg(all(any(target_arch = "aarch64", target_arch = "x86", target_arch = "x86_64"), not(target_env = "msvc")))'.dependencies]
sha1 = { version = "0.10.0", optional = true, features = ["asm"] }

Clearly, x86_64-pc-windows-msvc should not enable the asm feature, instead it should only pull in the sha1 crate without additional features.

To me this looks like something cargo gets wrong, but maybe I am just not getting how dependencies work with more complex per-target overrides.

My recommendation, in the meantime, is to use max-performance-safe for these platforms, and maybe do that based on target specific dependencies in cargo-binstall instead?

Some other strangeness is that CI already builds that platform - it will install the max version of gix which activates max-performance as well, and there it works as if the target-specific settings in gix-features do work, sometimes. The only difference I see is that it builds with --debug instead of --release.

When looking at the Cargo.toml of the sha1 crate, I see that they also try to make that available only on targets that are known to work, so something really isn't working as expected there, sometimes.

Maybe an avenue would be to understand why my CI seems to work on that environment, whereas the one at cargo-binstall doesn't? What's the difference, ideally this can be reproduced here. In the meantime, I think it's worth trying to workaround in cargo-binstall as well by selectively using max-performance-safe instead?

[NobodyXu]

Clearly, x86_64-pc-windows-msvc should not enable the asm feature, instead it should only pull in the sha1 crate without additional features.

Yeah I think this could be a bug in cargo.

My recommendation, in the meantime, is to use max-performance-safe for these platforms, and maybe do that based on target specific dependencies in cargo-binstall instead?

I agree, though I would probably still turns on flate2/zlib-ng since in cargo-bins/cargo-binstall#1184 we turned on support for custom registry and test it by pulling the crates.io git index.

It took much longer than I'd imagine, on my M1 it took at least 2m and on CI it took 11m.

[NobodyXu]

Some other strangeness is that CI already builds that platform - it will install the max version of gix which activates max-performance as well, and there it works as if the target-specific settings in gix-features do work, sometimes. The only difference I see is that it builds with --debug instead of --release.

Yeah that's really strange, because our CI also build binstall in debug and release and they both failed.

When looking at the Cargo.toml of the sha1 crate, I see that they also try to make that available only on targets that are known to work, so something really isn't working as expected there, sometimes.

Maybe an avenue would be to understand why my CI seems to work on that environment, whereas the one at cargo-binstall doesn't? What's the difference, ideally this can be reproduced here. In the meantime, I think it's worth trying to workaround in cargo-binstall as well by selectively using max-performance-safe instead?

Yeah, I will have another look at it once I'm free, it's really really strange.

[Byron]

It took much longer than I'd imagine, on my M1 it took at least 2m and on CI it took 11m.

You mean shallow cloning took 2m and 11m respectively?

It should be much faster, even without zlib-ng I'd expect the time dominated by transfer, and not pack resolution.

❯ gix clone --depth 1 https://github.com/rust-lang/crates.io-index --bare
 08:58:05 read pack done 82.7MB in 45.16s (1.8MB/s)
 08:58:05  indexing done 144.8k objects in 26.77s (5.4k objects/s)
 08:58:05 decompressing done 530.4MB in 26.77s (19.8MB/s)
 08:58:05     Resolving done 144.8k objects in 0.38s (382.3k objects/s)
 08:58:05      Decoding done 2.5GB in 0.38s (6.6GB/s)
 08:58:06 writing index file done 4.1MB in 0.01s (671.2MB/s)
 08:58:06  create index file done 144.8k objects in 27.17s (5.3k objects/s)
HEAD:refs/remotes/origin/HEAD (implicit)
        05fe54b35765137db5650c546a9efc5899cb378b HEAD symref-target:refs/heads/master -> refs/remotes/origin/HEAD [new]
+refs/heads/*:refs/remotes/origin/*
        05fe54b35765137db5650c546a9efc5899cb378b refs/heads/master -> refs/remotes/origin/master [new]

git ( master) [?] via 🐍 took 46s
❯ gix clone https://github.com/rust-lang/crates.io-index --bare crates-io-full
 08:58:52 read pack done 310.7MB in 29.54s (10.5MB/s)
 08:58:52  indexing done 531.1k objects in 29.17s (18.2k objects/s)
 08:58:52 decompressing done 1.8GB in 29.17s (61.9MB/s)
 08:58:54     Resolving done 531.1k objects in 1.59s (333.7k objects/s)
 08:58:54      Decoding done 18.4GB in 1.59s (11.6GB/s)
 08:58:54 writing index file done 14.9MB in 0.02s (638.8MB/s)
 08:58:54  create index file done 531.1k objects in 30.85s (17.2k objects/s)
HEAD:refs/remotes/origin/HEAD (implicit)
        05fe54b35765137db5650c546a9efc5899cb378b HEAD symref-target:refs/heads/master -> refs/remotes/origin/HEAD [new]
+refs/heads/*:refs/remotes/origin/*
        05fe54b35765137db5650c546a9efc5899cb378b refs/heads/master -> refs/remotes/origin/master [new]

Above it shows that --depth 1 is actually slower than a full clone as the remote has more work.

[NobodyXu]

You mean shallow cloning took 2m and 11m respectively?

Yes, I think it could be because the test is run in debug mode.

Even in release, cargo-binstall is compiled using O2, so a better option might be to always compile gix and its dependencies with O3, probably more reliable than max-performance.

[NobodyXu]

Yeah setting:

# Set the default for dependencies on debug.
[profile.dev.package."*"]
opt-level = "z"

speedup the cloning to 2m.
I guess it's time for binstall to use O2 or even O3 given that gix really need all that optimization.

[NobodyXu]

Yeah, I will have another look at it once I'm free, it's really really strange.

Building gix directly with:

cargo build --features max-performance,blocking-http-transport-reqwest-rust-tls --target x86_64-pc-windows-msvc

always succeed in dev and release mode, with or without blocking-http-transport-reqwest-rust-tls.

But somehow when building from cargo-binstall, it failed due to sha1-asm.

[NobodyXu]

Running cargo tree -i sha1-asm --target x86_64-pc-windows-msvc in cargo-binstall gives:

sha1-asm v0.5.1
└── sha1 v0.10.5
    └── gix-features v0.31.1
        ├── gix v0.48.0
        │   └── binstalk v0.13.0 (/Users/nobodyxu/Dev/cargo-binstall/crates/binstalk)
        ├── gix-commitgraph v0.17.1
        │   ├── gix v0.48.0 (*)
        │   ├── gix-negotiate v0.4.0
        │   │   └── gix v0.48.0 (*)
        │   ├── gix-revwalk v0.3.0
        │   │   ├── gix-negotiate v0.4.0 (*)
        │   │   ├── gix-revision v0.17.0
        │   │   │   ├── gix v0.48.0 (*)
        │   │   │   └── gix-refspec v0.13.0
        │   │   │       └── gix v0.48.0 (*)
        │   │   └── gix-traverse v0.29.0
        │   │       ├── gix v0.48.0 (*)
        │   │       ├── gix-index v0.20.0
        │   │       │   ├── gix v0.48.0 (*)
        │   │       │   └── gix-worktree v0.21.1
        │   │       │       └── gix v0.48.0 (*)
        │   │       └── gix-pack v0.39.1
        │   │           ├── gix v0.48.0 (*)
        │   │           └── gix-odb v0.49.1
        │   │               └── gix v0.48.0 (*)
        │   └── gix-traverse v0.29.0 (*)
        ├── gix-config v0.25.1
        │   └── gix v0.48.0 (*)
        ├── gix-fs v0.3.0
        │   ├── gix v0.48.0 (*)
        │   ├── gix-ref v0.32.1
        │   │   ├── gix v0.48.0 (*)
        │   │   ├── gix-config v0.25.1 (*)
        │   │   └── gix-discover v0.21.1
        │   │       └── gix v0.48.0 (*)
        │   ├── gix-tempfile v7.0.0
        │   │   ├── gix v0.48.0 (*)
        │   │   ├── gix-lock v7.0.1
        │   │   │   ├── gix v0.48.0 (*)
        │   │   │   ├── gix-index v0.20.0 (*)
        │   │   │   └── gix-ref v0.32.1 (*)
        │   │   ├── gix-pack v0.39.1 (*)
        │   │   └── gix-ref v0.32.1 (*)
        │   └── gix-worktree v0.21.1 (*)
        ├── gix-glob v0.9.1
        │   ├── gix v0.48.0 (*)
        │   ├── gix-attributes v0.14.1
        │   │   ├── gix v0.48.0 (*)
        │   │   └── gix-worktree v0.21.1 (*)
        │   ├── gix-config v0.25.1 (*)
        │   ├── gix-ignore v0.4.1
        │   │   ├── gix v0.48.0 (*)
        │   │   └── gix-worktree v0.21.1 (*)
        │   └── gix-worktree v0.21.1 (*)
        ├── gix-index v0.20.0 (*)
        ├── gix-object v0.32.0
        │   ├── gix v0.48.0 (*)
        │   ├── gix-diff v0.32.0
        │   │   ├── gix v0.48.0 (*)
        │   │   └── gix-pack v0.39.1 (*)
        │   ├── gix-index v0.20.0 (*)
        │   ├── gix-negotiate v0.4.0 (*)
        │   ├── gix-odb v0.49.1 (*)
        │   ├── gix-pack v0.39.1 (*)
        │   ├── gix-ref v0.32.1 (*)
        │   ├── gix-revision v0.17.0 (*)
        │   ├── gix-revwalk v0.3.0 (*)
        │   ├── gix-traverse v0.29.0 (*)
        │   └── gix-worktree v0.21.1 (*)
        ├── gix-odb v0.49.1 (*)
        ├── gix-pack v0.39.1 (*)
        ├── gix-protocol v0.35.0
        │   └── gix v0.48.0 (*)
        ├── gix-ref v0.32.1 (*)
        ├── gix-transport v0.33.1
        │   ├── gix v0.48.0 (*)
        │   └── gix-protocol v0.35.0 (*)
        ├── gix-url v0.20.1
        │   ├── gix v0.48.0 (*)
        │   ├── gix-credentials v0.16.1
        │   │   ├── gix v0.48.0 (*)
        │   │   ├── gix-protocol v0.35.0 (*)
        │   │   └── gix-transport v0.33.1 (*)
        │   └── gix-transport v0.33.1 (*)
        └── gix-worktree v0.21.1 (*)

But doing the same in gix gives:

$ cargo tree --target x86_64-pc-windows-msvc --features max-performance -i sha1-asm
warning: nothing to print.

To find dependencies that require specific target platforms, try to use option `--target all` first, and t
hen narrow your search scope accordingly.

[Byron]

Maybe there is a way to narrow down the cause of this by adding gix to a proxy-manifest that is freshly initialized, to see if it still somehow gets sha1-asm activated. If not, one can probably try to make the proxy more similar to the actual cargo-binstall manifest to learn what's causing this.

[NobodyXu]

Maybe there is a way to narrow down the cause of this by adding gix to a proxy-manifest that is freshly initialized, to see if it still somehow gets sha1-asm activated. If not, one can probably try to make the proxy more similar to the actual cargo-binstall manifest to learn what's causing this.

I am trying to do this, but it's really hard to setup env on windows.

Specifically, I have problem installing clang, I installed the clang component in vs installer but still doesn't work.