curl: enable tests

[risicle]

Description of changes

Previous comment disabling tests suggested these were too expensive. I find they take about 15 minutes, which I think is acceptable for such an important package (especially one we regularly patch). One test fails on pkgsMusl (tested nixos x86_64) and I've disabled it there, presumably some slight name resolution behaviour differences.

Some tests still skipped due to missing dependencies, but it's unlikely we want to add these build-time dependencies to such an early-stage package.

Tested curl and curlFull on indicated platforms, along with pkgsMusl.curl, pkgsStatic.curl & pkgsi686Linux.curl.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 22.05 Release Notes (or backporting 21.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[Mindavi]

I do agree that it's useful to run tests. Still kinda have a feeling that if we have (at some point) a dozen packages in the bootstrapping that all take 15 minutes, it'll severely increase the time to build stdenv for everyone. Also because they'll probably all build sequential too.

I wonder how much issues we'll catch with the tests, and also if we may want to run the tests separately from the build (to prevent stalling other subsequent builds during the long testing phase).

Just some thoughts, feel free to ignore :)

[mohe2015]

I wonder how much issues we'll catch with the tests, and also if we may want to run the tests separately from the build (to prevent stalling other subsequent builds during the long testing phase).

If at some point (I don't think this is currently the case) the test phase is fully separated from the build phase wouldn't this make sense for every package?

[risicle]

I do agree with you @Mindavi about concerns over these tests causing problems, and in some cases (see qemu) I've initially only enabled tests in a variant included in the passthru.tests. The biggest problem I see with this though, is that fails to properly deal with a package's numerous variants because of the self-referential problems passthru.tests has - and how we usually address them - i.e. allow callPackage to feed in the package from the top-level nixpkgs fixed-point. In this case only the vanilla package ever gets tested (let's not go into the semantics of myCurlVariant.tests therefore being misleading).

As for the time spent, I tend to think more about the potential time saved in catching problems early.

[lovek323]

I agree, it would be awesome to have tests for this package, but it failed when I tested it on aarch64-darwin.

nix-build -A curl && ./result-bin/bin/curl https://www.icanhazip.com && unlink ./result-bin
nix-build -A curlFull && ./result-bin/bin/curl https://www.icanhazip.com && unlink ./result-bin

Let me know if there's a better way to test these things, I'm keen to learn more.

Failures for aarch64-darwin:

TESTFAIL: These test cases failed: 1453

make[1]: *** [Makefile:807: quiet-test] Error 1
make[1]: Leaving directory '/private/tmp/nix-build-curl-7.82.0.drv-0/curl-7.82.0/tests'
make: *** [Makefile:1676: test] Error 2
error: builder for '/nix/store/zjgza82mkllgc9dnfmvxxvc2w3fxr06x-curl-7.82.0.drv' failed with exit code 2;
       last 10 log lines:
       > Use of uninitialized value $USER in substitution iterator at ./runtests.pl line 3416.
       > Use of uninitialized value $USER in substitution iterator at ./runtests.pl line 3416.
       > TESTDONE: 1474 tests were considered during 415 seconds.
       > TESTDONE: 97 tests out of 98 reported OK: 98%
       >
       > TESTFAIL: These test cases failed: 1453
       >
       > make[1]: *** [Makefile:807: quiet-test] Error 1
       > make[1]: Leaving directory '/private/tmp/nix-build-curl-7.82.0.drv-0/curl-7.82.0/tests'
       > make: *** [Makefile:1676: test] Error 2
       For full logs, run 'nix log /nix/store/zjgza82mkllgc9dnfmvxxvc2w3fxr06x-curl-7.82.0.drv'.
error: 1 dependencies of derivation '/nix/store/gyqcbzp0sbg3impay44580xmish9lnvj-bootstrap-stage2-stdenv-darwin.drv' failed to build
error: 1 dependencies of derivation '/nix/store/kwgg03wbrqaq908h4r5chxdfgi6w7nv7-bootstrap-stage2-stdenv-darwin.drv' failed to build
error: 1 dependencies of derivation '/nix/store/42f2ijq4rsjmm9jj7b7pfsipwp5n4d9k-bootstrap-stage3-stdenv-darwin.drv' failed to build
error: 1 dependencies of derivation '/nix/store/dri4h73yn430nah6zgv69vxq9yg36nai-bootstrap-stage3-stdenv-darwin.drv' failed to build
error: 1 dependencies of derivation '/nix/store/iz7sy49myhyq2mnnwkak9bk1ds79brxb-adv_cmds-119.drv' failed to build
error: 1 dependencies of derivation '/nix/store/9qfb59hqlc4bysgyag37ggzj1317na8v-bash-5.1-p16.drv' failed to build
error: 1 dependencies of derivation '/nix/store/y8hw0czbnlmhqk9gi1fbrqk74zahmr4n-bzip2-1.0.6.0.2.drv' failed to build
error: 1 dependencies of derivation '/nix/store/i8fkkvjnykvag0haz2j7ldgxp6nbpzfv-cctools-binutils-darwin-949.0.1.drv' failed to build
error: 1 dependencies of derivation '/nix/store/7m226610nyzz4g59wkpap7n8sn11wg66-clang-wrapper-11.1.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/8q7xla0ns5f1xkcddnrsjm8vgqcgkr5c-coreutils-9.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/l8qy0s5pmighwk49ghcb6qi4qh0584qx-fix-darwin-dylib-names-hook.drv' failed to build
error: 1 dependencies of derivation '/nix/store/8dyyibsl9cm7pynd1626hgd3ifzxqjjs-gnugrep-3.7.drv' failed to build
error: 1 dependencies of derivation '/nix/store/8v3rrbbf0izlcgxkhrsk8appbvzh0rjx-libffi-3.4.2.drv' failed to build
error: 1 dependencies of derivation '/nix/store/8w76sjflz5pr1fcyazlivpmkh176rl9s-libiconv-50.drv' failed to build
error: 1 dependencies of derivation '/nix/store/1y03jc6bn3h8y5xrqiwm23r4hsnxr8xl-libxml2-2.9.13.drv' failed to build
error: 1 dependencies of derivation '/nix/store/lrl4j63vmk7vz6x5r9swcgag8gkgjpby-llvm-11.1.0.drv' failed to build
error: 1 dependencies of derivation '/nix/store/401mvb9q1bfnb2qh1xn5mipr76k5z5da-ncurses-6.3.drv' failed to build
error: 1 dependencies of derivation '/nix/store/hlyf39f4rrbd8bvgbr1f8hk9j1drq6n1-xz-5.2.5.drv' failed to build
error: 1 dependencies of derivation '/nix/store/qb71sfv8ydh1ba3csklvlg0ihg9c20cg-zlib-1.2.12.drv' failed to build
error: 1 dependencies of derivation '/nix/store/1bmzp9j42dwjryvvn3gsfgfkr03xvi53-curl-7.82.0.drv' failed to build

[risicle]

On the arch64-darwin system I have access to test 1086 fails (though I suspect that is related to this specific machine's setup), but not 1453. Are you able to give me the output around the point 1453 fails?

p.s. do you have the sandbox enabled & what os version are you running? I'm running 12.3.1.

[lovek323]

p.s. do you have the sandbox enabled & what os version are you running? I'm running 12.3.1.

@risicle I'm using nix-darwin, so I have sandbox enabled in this way:

nix = {
  useSandbox = true;
}

in my ~/.nixpkgs/darwin-configuration.nix file.

I'm running 12.4 Beta (21F5058e), so that may cause some different outcomes as well.

[lovek323]

I get the following output for the failing test:

env: python3: No such file or directory
== Contents of files in the log/ dir after test 1450
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1450
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1450
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1450
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1450
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1450
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1450
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1450
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1450
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1450
RUN: failed to start the DICT server
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1451
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1451
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1451
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1451
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1451
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1451
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1451
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1451
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1451
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1451
RUN: failed to start the SMB server
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1452
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1452
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1452
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1452
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1452
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1452
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1452
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1452
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1452
env: python3: No such file or directory
== Contents of files in the log/ dir after test 1452
RUN: failed to start the TELNET server
test 1453... exit FAILED

It's complaining about a few things I could try to fix such as a missing python3 command and being unable to start the TELNET server.

[risicle]

Don't worry about the complaining-about-missing-things - it seems to sensibly skip (or ignore the results) of any tests it can't setup correctly. It's purely the test 1453... exit FAILED message that's interesting here (though clearly not very useful). It doesn't print anything immediately afterward does it?

Anyway, if it fails on the sandboxed build we should probably skip that test on darwin. FWIW most darwin users run without the sandbox because it's so broken. Hydra doesn't even build with the sandbox enabled on darwin.

[lovek323]

Yeah, the next line is just the next test starting.

[Mindavi]

@ofborg build curl

[Mindavi]

Since this isn't a package upgrade, I guess we can consider it a non-breaking change. If any tests start failing or being flaky they should be easy to disable. Here goes nothing.

If the test times prove to be an issue we can always disable tests again.

[panayiotis]

On the aarch64-darwin system I have access to test 1086 fails...

I have the same issue on aarch64-darwin, test 1086 fails with:

test 1086...
1086: protocol FAILED!
There was no content at all in the file log/server.input.
Server glitch? Total curl failure? Returned: 28

When I rm the aforementioned test, the derivation builds.

[Mindavi]

Guess that test needs to be disabled too then.

[Mindavi]

To reduce test time it might also help to remove tests that take, say, >= 5s or so. There are some that take 10s or longer, it may or may not be worth disabling some of those to still get reasonable coverage but also more reasonable build times.

[risicle]

Pushed 954efdf to staging-next

[risicle]

re: the speed, the most fruitful thing could be finding the tests which take some time to do precisely nothing because they fail to find a tool they need and figuring out why.