gitoxide auth tests sporadically failing

[ehuss]

Some of the gitoxide tests seem to be randomly failing in CI:

• Check publish_to_alt_registry publish content #11799 (comment)
• Bump crates-io to 0.36.0 #11820 (comment)

The errors look like:

---- git_auth::http_auth_offered stdout ----
running `/Users/runner/work/cargo/cargo/target/debug/cargo build -v`
running `/Users/runner/work/cargo/cargo/target/debug/cargo check`
thread '<unnamed>' panicked at 'assertion failed: `(left == right)`
  left: `{}`,
 right: `{"GET /foo/bar/info/refs?service=git-upload-pack HTTP/1.1", "Authorization: Basic Zm9vOmJhcg==", "Accept: */*"}`', tests/testsuite/git_auth.rs:63:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'git_auth::http_auth_offered' panicked at '
test failed running `/Users/runner/work/cargo/cargo/target/debug/cargo check`
error: expected to find:
    Updating git repository `http://127.0.0.1:50629/foo/bar`
error: failed to get `bar` as a dependency of package `foo v0.0.1 [..]`

Caused by:
  failed to load source for dependency `bar`

Caused by:
  Unable to update http://127.0.0.1:50629/foo/bar

Caused by:
  failed to clone into: [..]

Caused by:
  failed to authenticate when downloading repository

  * attempted to find username/password via `credential.helper`, but [..]

  if the git CLI succeeds then `net.git-fetch-with-cli` may help here
  https://[..]

Caused by:


did not find in output:
    Updating git repository `http://127.0.0.1:50629/foo/bar`
error: failed to get `bar` as a dependency of package `foo v0.0.1 (/Users/runner/work/cargo/cargo/target/tmp/cit/testsuite/git_auth/http_auth_offered/foo)`

Caused by:
  failed to load source for dependency `bar`

Caused by:
  Unable to update http://127.0.0.1:50629/foo/bar

Caused by:
  failed to clone into: /Users/runner/work/cargo/cargo/target/tmp/cit/testsuite/git_auth/http_auth_offered/home/.cargo/git/db/bar-71575e16952ccc70

Caused by:
  failed to authenticate when downloading repository

  * attempted to find username/password via git's `credential.helper` support, but failed

  if the git CLI succeeds then `net.git-fetch-with-cli` may help here
  https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli

Caused by:
  Failed to obtain credentials

Caused by:
  An IO error occurred while communicating to the credentials helper

Caused by:
  Broken pipe (os error 32)
', tests/testsuite/git_auth.rs:163:10

---- git_auth::instead_of_url_printed stdout ----
running `/Users/runner/work/cargo/cargo/target/debug/cargo build -v`
running `/Users/runner/work/cargo/cargo/target/debug/cargo check`
thread '<unnamed>' panicked at 'assertion failed: `(left == right)`
  left: `{}`,
 right: `{"Authorization: Basic Zm9vOmJhcg==", "Accept: */*", "GET /foo/bar/info/refs?service=git-upload-pack HTTP/1.1"}`', tests/testsuite/git_auth.rs:63:9
thread 'git_auth::instead_of_url_printed' panicked at '
test failed running `/Users/runner/work/cargo/cargo/target/debug/cargo check`
error: stderr did not match:
1   1         Updating git repository `https://foo.bar/foo/bar`
2   2     error: failed to get `bar` as a dependency of package `foo [..]`
3   3     
4   4     Caused by:
5   5       failed to load source for dependency `bar`
6   6     
7   7     Caused by:
8   8       Unable to update https://foo.bar/foo/bar
9   9     
10  10    Caused by:
11  11      failed to clone into: [..]
12  12    
13  13    Caused by:
14  14      failed to authenticate when downloading repository: http://127.0.0.1:50632/foo/bar
15  15    
16       -  * attempted to find username/password via `credential.helper`, but maybe the found credentials were incorrect
    16   +  * attempted to find username/password via git's `credential.helper` support, but failed
17  17    
18  18      if the git CLI succeeds then `net.git-fetch-with-cli` may help here
19  19      https://[..]
20  20    
21  21    Caused by:
22  22      [..]
23  23    
24  24    Caused by:
25  25      [..]
    26   +
    27   +Caused by:
    28   +  Broken pipe (os error 32)


---- git_auth::instead_of_url_printed stdout ----
running `/home/runner/work/cargo/cargo/target/debug/cargo build -v`
running `/home/runner/work/cargo/cargo/target/debug/cargo check`
thread '<unnamed>' panicked at 'assertion failed: `(left == right)`
  left: `{}`,
 right: `{"GET /foo/bar/info/refs?service=git-upload-pack HTTP/1.1", "Authorization: Basic Zm9vOmJhcg==", "Accept: */*"}`', tests/testsuite/git_auth.rs:63:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'git_auth::instead_of_url_printed' panicked at '
test failed running `/home/runner/work/cargo/cargo/target/debug/cargo check`
error: stderr did not match:
1   1         Updating git repository `https://foo.bar/foo/bar`
2   2     error: failed to get `bar` as a dependency of package `foo [..]`
3   3     
4   4     Caused by:
5   5       failed to load source for dependency `bar`
6   6     
7   7     Caused by:
8   8       Unable to update https://foo.bar/foo/bar
9   9     
10  10    Caused by:
11  11      failed to clone into: [..]
12  12    
13  13    Caused by:
14  14      failed to authenticate when downloading repository: http://127.0.0.1:43505/foo/bar
15  15    
16       -  * attempted to find username/password via `credential.helper`, but maybe the found credentials were incorrect
    16   +  * attempted to find username/password via git's `credential.helper` support, but failed
17  17    
18  18      if the git CLI succeeds then `net.git-fetch-with-cli` may help here
19  19      https://[..]
20  20    
21  21    Caused by:
22  22      [..]
23  23    
24  24    Caused by:
25  25      [..]
    26   +
    27   +Caused by:
    28   +  Broken pipe (os error 32)

[ehuss]

@Byron This seems to be happening fairly frequently. Would it be possible to ignore these tests or disable the gitoxide tests until this is fixed?

[Byron]

I think it should be possible to disable these tests only for gitoxide test runs and put fixing this on the list of things to do for an immediate remedy. I will submit a PR for that today.

For context, what happens here is that cargo configures a credential helper that it builds itself which merely echos credentials in a format expected by git. It should be impossible to not be able to read that in time and I don't know why the process can launch, but then somehow we fail to read its stdout.

What's even stranger is that git2 doesn't seem to have this problem, and all gitoxide does is to spawn a process and communicate through its stdin and out pipes. It's unclear to me how this program can fail at all. Maybe it didn't fail even, maybe it exited already and closed its stdin which we try to write, and gitoxide considers write failures whereas git2 does ignore write failures.

Let me also ignore write failures (just like git does) and submit a PR with a gix update if necessary.

[Byron]

This is more complicated than it has any right to be, but here we are.

• Disable flaky auth tests when gitoxide runs them #11830 - ignore flaky tests for now without any side-effects.
• [BLOCKED] upgrade gix to v0.41 to fix flaky auth tests. #11831 - can only be merged once tempfile 3.4 doesn't break builds anymore.

Are there better ways to do this? This is the only way I see unfortunately.

[weihanglo]

This was fixed by #11840 apparently, as it hasn't happened for a long while. Closing.

[ehuss]

@weihanglo Doesn't #11830 need to be reverted?

[weihanglo]

Hmm… we didn't?

[weihanglo]

The old failure should be gone but there is a new one on Windows. See #13116 (comment).

Feel free to take over it.

[Byron]

I was actually convinced that something was indeed reverted in the shallow-clones PR, but judging by this PR on Windows strange things are still happening.

To me it seems that it would already be an improvement to enable the two disabled tests on all platforms but windows for now, and I can work on a way to let library users control what happens to stderr of launched programs. Most of the time, it's set to inherit, which definitely isn't helpful for the tests. Maybe there is also a way to change the timeout which seems to cause these extra messages.

I will probably submit a PR for this tomorrow as the issue on windows seems fixable, so we can finally close this issue here.

[weihanglo]

Sounds good to me for either disabling or fixing the test on Windows. Thank you for taking a look at it!

[Byron]

Great! I gave it a quick shot but wasn't lucky enough for my 'fix' to fully work. This means gitoxide needs an adjustment to accommodate, but I hope to make that happen tomorrow then. With it, that issue should be permanently fixed though, which probably is the most desirable kind of fix anyway :).

[weihanglo]

Reopen. See #13129

[Byron]

The quick-shot must have missed :/. Admittedly I was flying blind and didn't validate that it works locally. I will try again and see what the problem is (this time with validation), as right now is a good time to try to tackle it as I can still make a point release of whichever crates are affected in the fix.

[weihanglo]

I feel confident to close this via #13130 this time.