Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Apply GitHub fast path even for partial hashes #10807

Merged
merged 1 commit into from
Aug 24, 2022

Conversation

dtolnay
Copy link
Member

@dtolnay dtolnay commented Jun 30, 2022

What does this PR try to resolve?

As flagged in #10079 (comment), it's not great to assume that git SHAs would always be 40-character hex strings. In the future they will be longer.

Git is on a long-term trajectory to move to SHA256 hashes (current status). I suppose when that becomes available/the default it's possible for a 40-digit hex-encoded hash not to be the full hash. Will this fail for that case?

The implementation from #10079 fails in that situation because it turns dependencies of the form { git = "…", rev = "[…40 hex…]" } into fetches with a refspec +[…40 hex…]:refs/commit/[…40 hex…]. That only works if the 40 hex digits are the full long hash of your commit. If it's really a prefix ("short hash") of a 64-hex-digit SHA-256 commit hash, you'd get a failure that resembles:

error: failed to get `dependency` as a dependency of package `repro v0.0.0`

Caused by:
  failed to load source for dependency `dependency`

Caused by:
  Unable to update https://github.com/rust-lang/dependency?rev=b30694b4d9b29141298870b7993e9aee10940524

Caused by:
  revspec 'b30694b4d9b29141298870b7993e9aee10940524' not found; class=Reference (4); code=NotFound (-3)

This PR updates the implementation so that Cargo will curl GitHub to get a resolved long commit hash even if the rev specified for the git dependency in Cargo.toml already looks like a SHA-1 long hash.

Performance considerations

⛔ This reverses a (questionable, negligible) benefit of #10079 of skipping the curl when rev is a long hash and is not already present in the local clone. These curls take 200-250ms on my machine.

🟰 We retain the much larger benefit of #10079 which comes from being able to precisely fetch a single rev, instead of fetching all branches and tags in the upstream repo and hoping to find the rev somewhere in there. This accounts for the entire performance difference explained in the summary of that PR.

🟰 We still skip the curl when rev is a long hash of a commit that is already previously fetched.

🥳 After this PR, we also curl and hit fast path when rev is a short hash of some upstream commit. For example { git = "https://github.com/rust-lang/cargo", rev = "b30694b4d9" } would previously have done the download-all-branches-and-tags codepath because b30694b4d9 is not a long hash. After this PR, the curl to GitHub informs us that b30694b4d9 resolves to the long hash b30694b4d9b29141298870b7993e9aee10940524, and we download just that commit instead of all-branches-and-tags.

How should we test and review this PR?

I tested with the following dependency specs, using /path/to/target/release/cargo generate-lockfile.

# Before: slow path (fetch all branches and tags; 70K git objects)
# After: fast path (20K git objects)
cargo = { git = "https://github.com/rust-lang/cargo", rev = "b30694b4d9b2914129" }
# Before and after: fast path
cargo = { git = "https://github.com/rust-lang/cargo", rev = "b30694b4d9b29141298870b7993e9aee10940524" }
# Before and after: fast path
cargo = { git = "https://github.com/rust-lang/cargo", rev = "refs/heads/rust-1.14.0" }
# Before and after: same error "revspec 'rust-1.14.0' not found"
# You are supposed to use `branch = "rust-1.14.0"`, this is not considered a `rev`
cargo = { git = "https://github.com/rust-lang/cargo", rev = "rust-1.14.0" }

I made sure these all work both with and without rm -rf ~/.cargo/git/db/cargo-* ~/.cargo/git/checkouts/cargo-* in between each cargo invocation.

FYI @djc

@rust-highfive
Copy link

r? @ehuss

(rust-highfive has picked a reviewer for you, use r? to override)

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jun 30, 2022
Copy link
Member

@weihanglo weihanglo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable to me. Thanks!

src/cargo/sources/git/utils.rs Show resolved Hide resolved
@Eh2406
Copy link
Contributor

Eh2406 commented Jul 21, 2022

I find the questions of longer SHAs confusing and a little premature. What is the timeline for git to support them? GitHub? LibGit2? What is the time pressure on Cargo? Are other aspects of Cargo ready for this change?

However, when looking at the code this seems like a much smaller change. We now read the result of https://api.github.com/repos/{}/{}/commits/{}, which we had previously just been querying for the status code, and use that for the follow-up fast path to GitHub. This maximizes the chance that we can use GitHubs fast path.

@est31
Copy link
Member

est31 commented Jul 21, 2022

This provides some reading on sha-256 progress in git: https://lwn.net/Articles/898522/

@dtolnay dtolnay changed the title SHA-256 support Apply GitHub fast path even for partial hashes Jul 21, 2022
@dtolnay
Copy link
Member Author

dtolnay commented Jul 21, 2022

Yeah this PR was poorly titled. This from the PR description is the actual main benefit, which SHA-256 is just a special case of:

🥳 After this PR, we also curl and hit fast path when rev is a short hash of some upstream commit. For example { git = "https://github.com/rust-lang/cargo", rev = "b30694b4d9" } would previously have done the download-all-branches-and-tags codepath because b30694b4d9 is not a long hash. After this PR, the curl to GitHub informs us that b30694b4d9 resolves to the long hash b30694b4d9b29141298870b7993e9aee10940524, and we download just that commit instead of all-branches-and-tags.

@weihanglo weihanglo added the T-cargo Team: Cargo label Aug 16, 2022
@weihanglo
Copy link
Member

weihanglo commented Aug 16, 2022

@rfcbot merge

This pull request reverts the benefits of #10079 of skipping curl for 40-characters hex.

OTOH it becomes future-proof when Git switches to SHA256, and also gains the benefit of skipping fetches for partial hashes when local objects already exist.

@rfcbot
Copy link
Collaborator

rfcbot commented Aug 16, 2022

Team member @weihanglo has proposed to merge this. The next step is review by the rest of the tagged team members:

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

@rfcbot rfcbot added proposed-final-comment-period An FCP proposal has started, but not yet signed off. disposition-merge FCP with intent to merge final-comment-period FCP — a period for last comments before action is taken labels Aug 16, 2022
@rfcbot
Copy link
Collaborator

rfcbot commented Aug 17, 2022

🔔 This is now entering its final comment period, as per the review above. 🔔

@rfcbot rfcbot removed the proposed-final-comment-period An FCP proposal has started, but not yet signed off. label Aug 17, 2022
@weihanglo
Copy link
Member

I feel like it's no harm to end the FCP earlier. Going to merge it now.

@bors r+

@bors
Copy link
Contributor

bors commented Aug 24, 2022

📌 Commit 36053de has been approved by weihanglo

It is now in the queue for this repository.

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 24, 2022
@bors
Copy link
Contributor

bors commented Aug 24, 2022

⌛ Testing commit 36053de with merge 9ded34a...

@bors
Copy link
Contributor

bors commented Aug 24, 2022

☀️ Test successful - checks-actions
Approved by: weihanglo
Pushing 9ded34a to master...

@bors bors merged commit 9ded34a into rust-lang:master Aug 24, 2022
@dtolnay dtolnay deleted the sha256 branch August 24, 2022 09:06
@rfcbot rfcbot added finished-final-comment-period FCP complete to-announce and removed final-comment-period FCP — a period for last comments before action is taken labels Aug 27, 2022
bors pushed a commit to rust-lang-ci/rust that referenced this pull request Aug 29, 2022
5 commits in 6da726708a4406f31f996d813790818dce837161..4ed54cecce3ce9ab6ff058781f4c8a500ee6b8b5
2022-08-23 21:39:56 +0000 to 2022-08-27 18:41:39 +0000
- doc: pause, for readability (rust-lang/cargo#11027)
- Bump git2 to 0.15 and libgit2-sys to 0.14 (rust-lang/cargo#11004)
- Fix typo (rust-lang/cargo#11025)
- Update cargo-toml-vs-cargo-lock.md (rust-lang/cargo#11021)
- Apply GitHub fast path even for partial hashes (rust-lang/cargo#10807)
bors added a commit to rust-lang-ci/rust that referenced this pull request Aug 29, 2022
…acrum

Update cargo

5 commits in 6da726708a4406f31f996d813790818dce837161..4ed54cecce3ce9ab6ff058781f4c8a500ee6b8b5
2022-08-23 21:39:56 +0000 to 2022-08-27 18:41:39 +0000
- doc: pause, for readability (rust-lang/cargo#11027)
- Bump git2 to 0.15 and libgit2-sys to 0.14 (rust-lang/cargo#11004)
- Fix typo (rust-lang/cargo#11025)
- Update cargo-toml-vs-cargo-lock.md (rust-lang/cargo#11021)
- Apply GitHub fast path even for partial hashes (rust-lang/cargo#10807)
@ehuss ehuss added this to the 1.65.0 milestone Aug 31, 2022
wip-sync pushed a commit to NetBSD/pkgsrc-wip that referenced this pull request Nov 27, 2022
Pkgsrc changes:
 * We now manage to build for mipsel-unknown-netbsd, but despite the
   target spec saying cpu = "mips3", the compiler manages to emit 64-bit
   instructions which cause "illegal instruction" error.  Will need more
   work.  The mipsel-unknown-netbsd entry is commentd out since there
   is no 1.64.0 bootstrap.
 * Managed to retain the build of aarch64_be, llvm needed a patch to
   avoid use of neon instructions in the BE case (llvm doesn't support
   use of neon in BE mode).  Ref. patch to
   src/llvm-project/llvm/lib/Support/BLAKE3/blake3_impl.h.
 * The minimum gcc version is now 7.x, and that includes the cross-compiler
   for the targets.  For i386 this also needs to /usr/include/gcc-7 include
   files in the target root, because immintrin.h from gcc 5 is not
   compatible with gcc 7.x.  This applies for the targets where we build
   against a root from netbsd-8 (sparc64, powerpc, i386), and files/gcc-wrap
   gets a hack for this.
 * Pick up tweak for -latomic inclusion from
   rust-lang/rust#104220
   and
   rust-lang/rust#104572
 * Retain ability to do 32-bit NetBSD, by changing from 64 to 32 bit
   types in library/std/src/sys/unix/thread_parker/netbsd.rs.
 * I've struggled a bit to get the "openssl-src" build with -latomic
   where it's needed.  I introduce "NetBSD-generic32" system type and
   use it for the NetBSD mipsel target.  There is another attempt to
   do the same in the patch to vendor/openssl-sys/build/main.rs.
 * Bump bootstraps to 1.64.0, checksum updates.

Upstream changes:

Version 1.65.0 (2022-11-03)
==========================

Language
--------
- [Error on `as` casts of enums with `#[non_exhaustive]` variants]
  (rust-lang/rust#92744)
- [Stabilize `let else`](rust-lang/rust#93628)
- [Stabilize generic associated types (GATs)]
  (rust-lang/rust#96709)
- [Add lints `let_underscore_drop`, `let_underscore_lock`, and
  `let_underscore_must_use` from Clippy]
  (rust-lang/rust#97739)
- [Stabilize `break`ing from arbitrary labeled blocks ("label-break-value")]
  (rust-lang/rust#99332)
- [Uninitialized integers, floats, and raw pointers are now considered
  immediate UB](rust-lang/rust#98919).
  Usage of `MaybeUninit` is the correct way to work with uninitialized
  memory.
- [Stabilize raw-dylib for Windows x86_64, aarch64, and thumbv7a]
  (rust-lang/rust#99916)
- [Do not allow `Drop` impl on foreign ADTs]
  (rust-lang/rust#99576)

Compiler
--------
- [Stabilize -Csplit-debuginfo on Linux]
  (rust-lang/rust#98051)
- [Use niche-filling optimization even when multiple variants have
  data] (rust-lang/rust#94075)
- [Associated type projections are now verified to be well-formed
  prior to resolving the underlying type]
  (rust-lang/rust#99217)
- [Stringify non-shorthand visibility correctly]
  (rust-lang/rust#100350)
- [Normalize struct field types when unsizing]
  (rust-lang/rust#101831)
- [Update to LLVM 15](rust-lang/rust#99464)
- [Fix aarch64 call abi to correctly zeroext when needed]
  (rust-lang/rust#97800)
- [debuginfo: Generalize C++-like encoding for enums]
  (rust-lang/rust#98393)
- [Add `special_module_name` lint]
  (rust-lang/rust#94467)
- [Add support for generating unique profraw files by default when
  using `-C instrument-coverage`]
  (rust-lang/rust#100384)
- [Allow dynamic linking for iOS/tvOS targets]
  (rust-lang/rust#100636)

New targets:
- [Add armv4t-none-eabi as a tier 3 target]
  (rust-lang/rust#100244)
- [Add powerpc64-unknown-openbsd and riscv64-unknown-openbsd as tier 3 targets]
  (rust-lang/rust#101025)
- Refer to Rust's [platform support page][platform-support-doc] for more
  information on Rust's tiered platform support.

Libraries
---------
- [Don't generate `PartialEq::ne` in derive(PartialEq)]
  (rust-lang/rust#98655)
- [Windows RNG: Use `BCRYPT_RNG_ALG_HANDLE` by default]
  (rust-lang/rust#101325)
- [Forbid mixing `System` with direct system allocator calls]
  (rust-lang/rust#101394)
- [Document no support for writing to non-blocking stdio/stderr]
  (rust-lang/rust#101416)
- [`std::layout::Layout` size must not overflow `isize::MAX` when
  rounded up to `align`](rust-lang/rust#95295)
  This also changes the safety conditions on
  `Layout::from_size_align_unchecked`.

Stabilized APIs
---------------
- [`std::backtrace::Backtrace`]
  (https://doc.rust-lang.org/stable/std/backtrace/struct.Backtrace.html)
- [`Bound::as_ref`]
  (https://doc.rust-lang.org/stable/std/ops/enum.Bound.html#method.as_ref)
- [`std::io::read_to_string`]
  (https://doc.rust-lang.org/stable/std/io/fn.read_to_string.html)
- [`<*const T>::cast_mut`]
  (https://doc.rust-lang.org/stable/std/primitive.pointer.html#method.cast_mut)
- [`<*mut T>::cast_const`]
  (https://doc.rust-lang.org/stable/std/primitive.pointer.html#method.cast_const)

These APIs are now stable in const contexts:
- [`<*const T>::offset_from`]
  (https://doc.rust-lang.org/stable/std/primitive.pointer.html#method.offset_from)
- [`<*mut T>::offset_from`]
  (https://doc.rust-lang.org/stable/std/primitive.pointer.html#method.offset_from)

Cargo
-----
- [Apply GitHub fast path even for partial hashes]
  (rust-lang/cargo#10807)
- [Do not add home bin path to PATH if it's already there]
  (rust-lang/cargo#11023)
- [Take priority into account within the pending queue]
  (rust-lang/cargo#11032).
  This slightly optimizes job scheduling by Cargo, with typically
  small improvements on larger crate graph builds.

Compatibility Notes
-------------------
- [`std::layout::Layout` size must not overflow `isize::MAX` when
  rounded up to `align`] (rust-lang/rust#95295).
  This also changes the safety conditions on
  `Layout::from_size_align_unchecked`.
- [`PollFn` now only implements `Unpin` if the closure is `Unpin`]
  (rust-lang/rust#102737).
  This is a possible breaking change if users were relying on the
  blanket unpin implementation.  See discussion on the PR for
  details of why this change was made.
- [Drop ExactSizeIterator impl from std::char::EscapeAscii]
  (rust-lang/rust#99880)
  This is a backwards-incompatible change to the standard library's
  surface area, but is unlikely to affect real world usage.
- [Do not consider a single repeated lifetime eligible for elision
  in the return type] (rust-lang/rust#103450)
  This behavior was unintentionally changed in 1.64.0, and this
  release reverts that change by making this an error again.
- [Reenable disabled early syntax gates as future-incompatibility
  lints] (rust-lang/rust#99935)
- [Update the minimum external LLVM to 13]
  (rust-lang/rust#100460)
- [Don't duplicate file descriptors into stdio fds]
  (rust-lang/rust#101426)
- [Sunset RLS](rust-lang/rust#100863)
- [Deny usage of `#![cfg_attr(..., crate_type = ...)]` to set the
  crate type] (rust-lang/rust#99784)
  This strengthens the forward compatibility lint
  deprecated_cfg_attr_crate_type_name to deny.
- [`llvm-has-rust-patches` allows setting the build system to treat
  the LLVM as having Rust-specific patches]
  (rust-lang/rust#101072)
  This option may need to be set for distributions that are building
  Rust with a patched LLVM via `llvm-config`, not the built-in
  LLVM.

Internal Changes
----------------

These changes do not affect any public interfaces of Rust, but they represent
significant improvements to the performance or internals of rustc and related
tools.

- [Add `x.sh` and `x.ps1` shell scripts]
  (rust-lang/rust#99992)
- [compiletest: use target cfg instead of hard-coded tables]
  (rust-lang/rust#100260)
- [Use object instead of LLVM for reading bitcode from rlibs]
  (rust-lang/rust#98100)
- [Enable MIR inlining for optimized compilations]
  (rust-lang/rust#91743)
  This provides a 3-10% improvement in compiletimes for real world
  crates. See [perf results]
  (https://perf.rust-lang.org/compare.html?start=aedf78e56b2279cc869962feac5153b6ba7001ed&end=0075bb4fad68e64b6d1be06bf2db366c30bc75e1&stat=instructions:u).
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Jan 23, 2023
Pkgsrc changes:
 * pkglint cleanups, bump bootstrap kits to 1.65.0.
 * New target: mipsel-unknown-netbsd, for cpu=mips32 with soft-float.
 * Managed to retain the build of aarch64_be, llvm needed a patch to
   avoid use of neon instructions in the BE case (llvm doesn't support
   use of neon in BE mode).  Ref. patch to
   src/llvm-project/llvm/lib/Support/BLAKE3/blake3_impl.h.
   Also submitted upstream of LLVM to the BLAKE3 maintainers.
 * The minimum gcc version is now 7.x, and that includes the
   cross-compiler for the targets.  For i386 this also needs to
   /usr/include/gcc-7 include files in the target root, because
   immintrin.h from gcc 5 is not compatible with gcc 7.x.  This
   applies for the targets where we build against a root from netbsd-8
   (sparc64, powerpc, i386), and files/gcc-wrap gets a hack for this.
 * Pick up tweak for -latomic inclusion from
   rust-lang/rust#104220
   and
   rust-lang/rust#104572
 * Retain ability to do 32-bit NetBSD, by changing from 64 to 32 bit
   types in library/std/src/sys/unix/thread_parker/netbsd.rs.
 * I've tried to get the "openssl-src" build with -latomic where it's
   needed.  I've introduced the "NetBSD-generic32" system type and use
   it for the NetBSD mipsel target.  There is another attempt to do
   the same in the patch to vendor/openssl-sys/build/main.rs.


Upstream changes:

Version 1.66.1 (2023-01-10)
===========================

- Added validation of SSH host keys for git URLs in Cargo
  ([CVE-2022-46176](https://www.cve.org/CVERecord?id=CVE-2022-46176))


Version 1.66.0 (2022-12-15)
===========================

Language
--------
- [Permit specifying explicit discriminants on all `repr(Int)`
  enums](rust-lang/rust#95710)
  ```rust
  #[repr(u8)]
  enum Foo {
      A(u8) = 0,
      B(i8) = 1,
      C(bool) = 42,
  }
  ```
- [Allow transmutes between the same type differing only in
  lifetimes](rust-lang/rust#101520)
- [Change constant evaluation errors from a deny-by-default lint to a
  hard error](rust-lang/rust#102091)
- [Trigger `must_use` on `impl Trait` for
  supertraits](rust-lang/rust#102287) This
  makes `impl ExactSizeIterator` respect the existing `#[must_use]`
  annotation on `Iterator`.
- [Allow `..X` and `..=X` in
  patterns](rust-lang/rust#102275)
- [Uplift `clippy::for_loops_over_fallibles` lint into
  rustc](rust-lang/rust#99696)
- [Stabilize `sym` operands in inline
  assembly](rust-lang/rust#103168)
- [Update to Unicode 15](rust-lang/rust#101912)
- [Opaque types no longer imply lifetime
  bounds](rust-lang/rust#95474) This is a
  soundness fix which may break code that was erroneously relying on this
  behavior.

Compiler
--------
- [Add armv5te-none-eabi and thumbv5te-none-eabi tier 3
  targets](rust-lang/rust#101329)
  - Refer to Rust's [platform support page][platform-support-doc] for
    more information on Rust's tiered platform support.
- [Add support for linking against macOS universal
  libraries](rust-lang/rust#98736)

Libraries
---------
- [Fix `#[derive(Default)]` on a generic `#[default]` enum adding
  unnecessary `Default`
  bounds](rust-lang/rust#101040)
- [Update to Unicode 15](rust-lang/rust#101821)

Stabilized APIs
---------------
- [`proc_macro::Span::source_text`](https://doc.rust-lang.org/stable/proc_macro/struct.Span.html#method.source_text)
- [`uX::{checked_add_signed, overflowing_add_signed,
  saturating_add_signed,
  wrapping_add_signed}`](https://doc.rust-lang.org/stable/std/primitive.u8.html#method.checked_add_signed)
- [`iX::{checked_add_unsigned, overflowing_add_unsigned,
  saturating_add_unsigned,
  wrapping_add_unsigned}`](https://doc.rust-lang.org/stable/std/primitive.i8.html#method.checked_add_unsigned)
- [`iX::{checked_sub_unsigned, overflowing_sub_unsigned,
  saturating_sub_unsigned,
  wrapping_sub_unsigned}`](https://doc.rust-lang.org/stable/std/primitive.i8.html#method.checked_sub_unsigned)
- [`BTreeSet::{first, last, pop_first,
  pop_last}`](https://doc.rust-lang.org/stable/std/collections/struct.BTreeSet.html#method.first)
- [`BTreeMap::{first_key_value, last_key_value, first_entry, last_entry,
  pop_first,
  pop_last}`](https://doc.rust-lang.org/stable/std/collections/struct.BTreeMap.html#method.first_key_value)
- [Add `AsFd` implementations for stdio lock types on
  WASI.](rust-lang/rust#101768)
- [`impl TryFrom<Vec<T>> for Box<[T;
  N]>`](https://doc.rust-lang.org/stable/std/boxed/struct.Box.html#impl-TryFrom%3CVec%3CT%2C%20Global%3E%3E-for-Box%3C%5BT%3B%20N%5D%2C%20Global%3E)
- [`core::hint::black_box`](https://doc.rust-lang.org/stable/std/hint/fn.black_box.html)
- [`Duration::try_from_secs_{f32,f64}`](https://doc.rust-lang.org/stable/std/time/struct.Duration.html#method.try_from_secs_f32)
- [`Option::unzip`](https://doc.rust-lang.org/stable/std/option/enum.Option.html#method.unzip)
- [`std::os::fd`](https://doc.rust-lang.org/stable/std/os/fd/index.html)

Rustdoc
-------
- [Add Rustdoc warning for invalid HTML tags in the
  documentation](rust-lang/rust#101720)

Cargo
-----
- [Added `cargo remove` to remove dependencies from
  Cargo.toml](https://doc.rust-lang.org/nightly/cargo/commands/cargo-remove.html)
- [`cargo publish` now waits for the new version to be downloadable
  before exiting](rust-lang/cargo#11062)

See [detailed release notes](https://github.com/rust-lang/cargo/blob/master/CHANGELOG.md#cargo-166-2022-12-15) for more.

Compatibility Notes
-------------------
- [Only apply `ProceduralMasquerade` hack to older versions of
  `rental`](rust-lang/rust#94063)
- [Don't export `__heap_base` and `__data_end` on
  wasm32-wasi.](rust-lang/rust#102385)
- [Don't export `__wasm_init_memory` on
  WebAssembly.](rust-lang/rust#102426)
- [Only export `__tls_*` on
  wasm32-unknown-unknown.](rust-lang/rust#102440)
- [Don't link to `libresolv` in libstd on
  Darwin](rust-lang/rust#102766)
- [Update libstd's libc to 0.2.135 (to make `libstd` no longer pull in
  `libiconv.dylib` on
  Darwin)](rust-lang/rust#103277)
- [Opaque types no longer imply lifetime
  bounds](rust-lang/rust#95474)
  This is a soundness fix which may break code that was erroneously
  relying on this behavior.
- [Make `order_dependent_trait_objects` show up in future-breakage
  reports](rust-lang/rust#102635)
- [Change std::process::Command spawning to default to inheriting the
  parent's signal mask](rust-lang/rust#101077)

Internal Changes
----------------

These changes do not affect any public interfaces of Rust, but they
represent significant improvements to the performance or internals of
rustc and related tools.

- [Enable BOLT for LLVM
  compilation](rust-lang/rust#94381)
- [Enable LTO for
  rustc_driver.so](rust-lang/rust#101403)


Version 1.65.0 (2022-11-03)
==========================

Language
--------
- [Error on `as` casts of enums with `#[non_exhaustive]` variants]
  (rust-lang/rust#92744)
- [Stabilize `let else`](rust-lang/rust#93628)
- [Stabilize generic associated types (GATs)]
  (rust-lang/rust#96709)
- [Add lints `let_underscore_drop`, `let_underscore_lock`, and
  `let_underscore_must_use` from Clippy]
  (rust-lang/rust#97739)
- [Stabilize `break`ing from arbitrary labeled blocks ("label-break-value")]
  (rust-lang/rust#99332)
- [Uninitialized integers, floats, and raw pointers are now considered
  immediate UB](rust-lang/rust#98919).
  Usage of `MaybeUninit` is the correct way to work with uninitialized
  memory.
- [Stabilize raw-dylib for Windows x86_64, aarch64, and thumbv7a]
  (rust-lang/rust#99916)
- [Do not allow `Drop` impl on foreign ADTs]
  (rust-lang/rust#99576)

Compiler
--------
- [Stabilize -Csplit-debuginfo on Linux]
  (rust-lang/rust#98051)
- [Use niche-filling optimization even when multiple variants have
  data] (rust-lang/rust#94075)
- [Associated type projections are now verified to be well-formed
  prior to resolving the underlying type]
  (rust-lang/rust#99217)
- [Stringify non-shorthand visibility correctly]
  (rust-lang/rust#100350)
- [Normalize struct field types when unsizing]
  (rust-lang/rust#101831)
- [Update to LLVM 15](rust-lang/rust#99464)
- [Fix aarch64 call abi to correctly zeroext when needed]
  (rust-lang/rust#97800)
- [debuginfo: Generalize C++-like encoding for enums]
  (rust-lang/rust#98393)
- [Add `special_module_name` lint]
  (rust-lang/rust#94467)
- [Add support for generating unique profraw files by default when
  using `-C instrument-coverage`]
  (rust-lang/rust#100384)
- [Allow dynamic linking for iOS/tvOS targets]
  (rust-lang/rust#100636)

New targets:
- [Add armv4t-none-eabi as a tier 3 target]
  (rust-lang/rust#100244)
- [Add powerpc64-unknown-openbsd and riscv64-unknown-openbsd as tier 3 targets]
  (rust-lang/rust#101025)
- Refer to Rust's [platform support page][platform-support-doc] for more
  information on Rust's tiered platform support.

Libraries
---------
- [Don't generate `PartialEq::ne` in derive(PartialEq)]
  (rust-lang/rust#98655)
- [Windows RNG: Use `BCRYPT_RNG_ALG_HANDLE` by default]
  (rust-lang/rust#101325)
- [Forbid mixing `System` with direct system allocator calls]
  (rust-lang/rust#101394)
- [Document no support for writing to non-blocking stdio/stderr]
  (rust-lang/rust#101416)
- [`std::layout::Layout` size must not overflow `isize::MAX` when
  rounded up to `align`](rust-lang/rust#95295)
  This also changes the safety conditions on
  `Layout::from_size_align_unchecked`.

Stabilized APIs
---------------
- [`std::backtrace::Backtrace`]
  (https://doc.rust-lang.org/stable/std/backtrace/struct.Backtrace.html)
- [`Bound::as_ref`]
  (https://doc.rust-lang.org/stable/std/ops/enum.Bound.html#method.as_ref)
- [`std::io::read_to_string`]
  (https://doc.rust-lang.org/stable/std/io/fn.read_to_string.html)
- [`<*const T>::cast_mut`]
  (https://doc.rust-lang.org/stable/std/primitive.pointer.html#method.cast_mut)
- [`<*mut T>::cast_const`]
  (https://doc.rust-lang.org/stable/std/primitive.pointer.html#method.cast_const)

These APIs are now stable in const contexts:
- [`<*const T>::offset_from`]
  (https://doc.rust-lang.org/stable/std/primitive.pointer.html#method.offset_from)
- [`<*mut T>::offset_from`]
  (https://doc.rust-lang.org/stable/std/primitive.pointer.html#method.offset_from)

Cargo
-----
- [Apply GitHub fast path even for partial hashes]
  (rust-lang/cargo#10807)
- [Do not add home bin path to PATH if it's already there]
  (rust-lang/cargo#11023)
- [Take priority into account within the pending queue]
  (rust-lang/cargo#11032).
  This slightly optimizes job scheduling by Cargo, with typically
  small improvements on larger crate graph builds.

Compatibility Notes
-------------------
- [`std::layout::Layout` size must not overflow `isize::MAX` when
  rounded up to `align`] (rust-lang/rust#95295).
  This also changes the safety conditions on
  `Layout::from_size_align_unchecked`.
- [`PollFn` now only implements `Unpin` if the closure is `Unpin`]
  (rust-lang/rust#102737).
  This is a possible breaking change if users were relying on the
  blanket unpin implementation.  See discussion on the PR for
  details of why this change was made.
- [Drop ExactSizeIterator impl from std::char::EscapeAscii]
  (rust-lang/rust#99880)
  This is a backwards-incompatible change to the standard library's
  surface area, but is unlikely to affect real world usage.
- [Do not consider a single repeated lifetime eligible for elision
  in the return type] (rust-lang/rust#103450)
  This behavior was unintentionally changed in 1.64.0, and this
  release reverts that change by making this an error again.
- [Reenable disabled early syntax gates as future-incompatibility
  lints] (rust-lang/rust#99935)
- [Update the minimum external LLVM to 13]
  (rust-lang/rust#100460)
- [Don't duplicate file descriptors into stdio fds]
  (rust-lang/rust#101426)
- [Sunset RLS](rust-lang/rust#100863)
- [Deny usage of `#![cfg_attr(..., crate_type = ...)]` to set the
  crate type] (rust-lang/rust#99784)
  This strengthens the forward compatibility lint
  deprecated_cfg_attr_crate_type_name to deny.
- [`llvm-has-rust-patches` allows setting the build system to treat
  the LLVM as having Rust-specific patches]
  (rust-lang/rust#101072)
  This option may need to be set for distributions that are building
  Rust with a patched LLVM via `llvm-config`, not the built-in
  LLVM.

Internal Changes
----------------

These changes do not affect any public interfaces of Rust, but they represent
significant improvements to the performance or internals of rustc and related
tools.

- [Add `x.sh` and `x.ps1` shell scripts]
  (rust-lang/rust#99992)
- [compiletest: use target cfg instead of hard-coded tables]
  (rust-lang/rust#100260)
- [Use object instead of LLVM for reading bitcode from rlibs]
  (rust-lang/rust#98100)
- [Enable MIR inlining for optimized compilations]
  (rust-lang/rust#91743)
  This provides a 3-10% improvement in compiletimes for real world
  crates. See [perf results]
  (https://perf.rust-lang.org/compare.html?start=aedf78e56b2279cc869962feac5153b6ba7001ed&end=0075bb4fad68e64b6d1be06bf2db366c30bc75e1&stat=instructions:u).
RalfJung pushed a commit to RalfJung/rust-analyzer that referenced this pull request Apr 20, 2024
Update cargo

5 commits in 6da726708a4406f31f996d813790818dce837161..4ed54cecce3ce9ab6ff058781f4c8a500ee6b8b5
2022-08-23 21:39:56 +0000 to 2022-08-27 18:41:39 +0000
- doc: pause, for readability (rust-lang/cargo#11027)
- Bump git2 to 0.15 and libgit2-sys to 0.14 (rust-lang/cargo#11004)
- Fix typo (rust-lang/cargo#11025)
- Update cargo-toml-vs-cargo-lock.md (rust-lang/cargo#11021)
- Apply GitHub fast path even for partial hashes (rust-lang/cargo#10807)
RalfJung pushed a commit to RalfJung/rust-analyzer that referenced this pull request Apr 27, 2024
Update cargo

5 commits in 6da726708a4406f31f996d813790818dce837161..4ed54cecce3ce9ab6ff058781f4c8a500ee6b8b5
2022-08-23 21:39:56 +0000 to 2022-08-27 18:41:39 +0000
- doc: pause, for readability (rust-lang/cargo#11027)
- Bump git2 to 0.15 and libgit2-sys to 0.14 (rust-lang/cargo#11004)
- Fix typo (rust-lang/cargo#11025)
- Update cargo-toml-vs-cargo-lock.md (rust-lang/cargo#11021)
- Apply GitHub fast path even for partial hashes (rust-lang/cargo#10807)
bors added a commit that referenced this pull request May 22, 2024
fetch specific commits even if the github fast path fails

### What does this PR try to resolve?

This PR fixes #13555, which describes a regression from 1.64.0 to 1.65.0 where the inability to fetch commit information from api.github.com (the "GitHub fast path") silently changes Cargo's behavior.

Cargo can fetch a specific Git commit from a remote without having to fetch all refs. Prior to #10807, this functionality required a repository hosted on github.com and providing the full commit hash (usually available from the Cargo.lock); after that change, any revision (including abbreviated revisions) that could be resolved by GitHub's API could be fetched directly. However, this logic requires the "GitHub fast path", which was not intended to be robust, to successfully return the resolved commit hash; if a client is currently rate-limited by api.github.com (very common in CI and shared cloud / corporate environments) this fails and Cargo falls back to fetching all refs.

Usually this is not noticeable. However, GitHub allows fetching commits that are related to the repository but not actually part of any of its refs, including commits pushed to a fork. This results in the same command working fine in some environments where api.github.com is accessible, and not working in other environments that are rate-limited, which is very confusing and difficult to debug.

This change adds another branch to cover the regression case: if we are going through the GitHub fast path with a full commit hash, return early indicating that we need to fetch it. (Previously: ~~when the GitHub fast path was unsuccessful, the user is not using the unstable shallow clone options, and we have a full commit hash and expect to be able to fetch it directly because we know it's a github.com repository.~~)

### How should we test and review this PR?

I have been testing this PR by temporarily adding a `0.0.0.0 api.github.com` entry to my `/etc/hosts`, which causes the GitHub fast path to always fail, then running:

```
target/release/cargo install --git https://github.com/haha-business/unstable-test-repo.git --rev c9040898c9183ddbb9402dcbf749ed06d6ea90ad
```

This refers to [a particular commit on a fork of the repo](iliana/unstable-test-repo@c904089) which won't be found by the fallback path or current Cargo.

**Note** that you will need to delete `~/.cargo/git/checkouts/unstable-test-repo-*` and `~/.cargo/git/db/unstable-test-repo-*` after a successful run with this change in order to reproduce the broken behavior of the current release.

I am having trouble getting the test suite to run at all on my system so I haven't experimented with writing a specific test for this case, but I probably should.

### Additional information

This uses the same logic as the unstable shallow clone support to detect if the revision is a full commit hash. This is not compatible with SHA-256 commit hashes; `git2::Oid` specifically expects a 40-character hexadecimal string. Given that the change introducing this bug was meant to future-proof SHA-256 support (despite only doing so for GitHub repositories), it might be good to make the logic more explicit within Cargo and allow either 40- or 64-character hex strings.

I wanted to keep this change focused on the regression fix, but in testing, pretty much every Git repository I could think of (including non-forges, like git.kernel.org and some repositories I host on my own infrastructure with cgit) supports fetching directly from a commit, so it would be ideal to eventually relax the GitHub requirement for this functionality. However, it would need some sort of fallback logic because I suspect the HTTP [dumb protocol](https://git-scm.com/book/en/v2/Git-Internals-Transfer-Protocols) doesn't support commit references, and I haven't researched when this functionality was added to the smart protocol.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
disposition-merge FCP with intent to merge finished-final-comment-period FCP complete S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-cargo Team: Cargo
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants