pdm: init at 1.12.6

[cpcloud]

Motivation for this change

Add pdm to nixpkgs.

This PR depends on a number of other PRs, which is why it's in draft mode:

1. python3Packages.pdm-pep517: init at 0.9.4 #155385
2. python3Packages.resolvelib: 0.5.5 -> 0.8.1 #155380 incorporated here
3. python3Packages.pythonfinder: init at 1.2.9 #155378
4. Python unstable 2022-01-13 #154969 merging into python-unstable

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 22.05 Release Notes (or backporting 21.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[cpcloud]

This will fail basic eval until the PRs mentioned in the description land in master.

[cpcloud]

I tested this branch against #154969, and things look good:

[mweinelt]

We could merge this into python-unstable, given that python-unstable will take at least 2-3 weeks to reach master.

[cpcloud]

@mweinelt That sounds good to me. How do I do that?

[mweinelt]

Rebase onto python-unstable, force-push … and at the same time change the target branch of this pull request to python-unstable in the upper right corner. If you succeed this will be a quiet change, if not you will likely ping a large number of people. 😄

[cpcloud]

Rebase onto python-unstable, force-push … and at the same time change the target branch of this pull request to python-unstable in the upper right corner. If you succeed this will be a quiet change, if not you will likely ping a large number of people. smile

Ha, well, I did this:

gh pr edit -B python-unstable && git push -f

in an attempt to do both as simultaneous as possible 😅 .

[jonringer]

do you mind doing:

common=$(git merge-base python-unstable staging)
git rebase --onto=$common HEAD^
git push ... ... --force

The package can be tested by doing git merge origin/python-unstable --no-merge --no-ff

[mweinelt]

Please reenable the tests you deemed slow - they are not connecting to the network. The test suite runs with pytest-xdist upstream, which we should also do to alleviate some of the slowness.

Also there is an initial patch for marking network tests in the upstream issue, that looked promising.

Posting my changes as a patch, since with over 400 commits it's annoying to find the correct change.

diff --git a/pkgs/tools/package-management/pdm/default.nix b/pkgs/tools/package-management/pdm/default.nix
index cf8b7d3f74a..e9db7f63fba 100644
--- a/pkgs/tools/package-management/pdm/default.nix
+++ b/pkgs/tools/package-management/pdm/default.nix
@@ -1,4 +1,4 @@
-{ lib, python3, fetchFromGitHub }:
+{ lib, python3, fetchFromGitHub, fetchurl }:
 let
   python = python3.override {
     # override resolvelib due to
@@ -39,7 +39,14 @@ buildPythonApplication rec {
   #
   # the patch is necessary because the fixture is creating a project and
   # doesn't appear to respect the settings in `$HOME`; possibly a bug upstream
-  patches = [ ./check-update.patch ];
+  patches = [
+    ./check-update.patch
+    (fetchurl {
+      # Mark test that require network access
+      url = "https://github.com/pdm-project/pdm/files/7911962/mark-network-tests.patch.txt";
+      hash = "sha256:1dizf9j3z7zk4lxvnszwx63xzd9r68f2iva5sszzf8s8na831dvd";
+    })
+  ];
 
   propagatedBuildInputs = [
     blinker
@@ -64,121 +71,28 @@ buildPythonApplication rec {
   checkInputs = [
     pytestCheckHook
     pytest-mock
+    pytest-xdist
+  ];
+
+  pytestFlagsArray = [
+    "--numprocesses $NIX_BUILD_CORES"
+    "-m 'not network'"
   ];
 
   preCheck = "HOME=$TMPDIR";
 
   disabledTests = [
-    # these may eventually succeed but are incredibly slow due to contacting
-    # the internet
-    "test_convert_requirements_file_without_name"
-    "test_add_dependency_from_multiple_parents"
-    "test_add_editable_package"
-    "test_add_package"
-    "test_add_package_with_mismatch_marker"
-    "test_add_with_dry_run"
-    "test_add_with_prerelease"
-    "test_build_src_package_by_include"
-    "test_editable_package_override_non_editable"
-    "test_find_candidates_from_find_links"
-    "test_freeze_dependencies_list"
-    "test_info_global_project"
-    "test_install_with_dry_run"
-    "test_install_with_lockfile"
-    "test_list_dependency_graph"
-    "test_list_dependency_graph_with_circular_forward"
-    "test_list_dependency_graph_with_circular_reverse"
-    "test_list_reverse_dependency_graph"
-    "test_load_multiple_plugings"
-    "test_old_entry_point_compatibility"
-    "test_parse_flit_project_metadata"
-    "test_parse_local_directory_metadata"
-    "test_parse_project_file_on_build_error"
-    "test_parse_project_file_on_build_error_no_dep"
-    "test_parse_project_file_on_build_error_with_extras"
-    "test_parse_vcs_metadata"
-    "test_post_lock_and_install_signals"
-    "test_remove_both_normal_and_editable_packages"
-    "test_remove_package"
-    "test_remove_package_exist_in_multi_groups"
-    "test_remove_package_no_sync"
-    "test_remove_package_not_exist"
-    "test_remove_package_with_dry_run"
-    "test_resolve_file_req_with_prerelease"
-    "test_resolve_local_and_named_requirement"
-    "test_resolve_two_extras_from_the_same_package"
-    "test_resolve_vcs_and_local_requirements"
-    "test_sync_clean_packages"
-    "test_sync_dry_run"
-    "test_sync_packages_with_group_all"
-    "test_sync_packages_with_groups"
-    "test_sync_production_packages"
-    "test_update_all_packages"
-    "test_update_dry_run"
-    "test_update_existing_package_with_prerelease"
-    "test_update_ignore_constraints"
-    "test_update_packages_with_top"
-    "test_update_top_packages_dry_run"
-    "test_update_with_package_and_groups_argument"
-    # requires the internet
-    "test_actual_list_freeze"
-    "test_add_cached_vcs_requirement"
-    "test_add_remote_package_url"
-    "test_basic_integration"
-    "test_build_distributions"
-    "test_build_ignoring_pip_environment"
-    "test_build_legacy_package"
-    "test_build_package"
-    "test_build_package_include"
-    "test_build_single_module"
-    "test_build_single_module_with_readme"
-    "test_build_src_package"
-    "test_build_with_config_settings"
-    "test_build_with_no_isolation"
-    "test_cache_egg_info_sdist"
-    "test_cache_vcs_immutable_revision"
-    "test_cli_build_with_config_settings"
-    "test_expand_project_root_in_url"
-    "test_extras_warning"
-    "test_hash_cache"
-    "test_install_wheel_with_cache"
-    "test_install_wheel_with_data_scripts"
-    "test_install_wheel_with_inconsistent_dist_info"
-    "test_install_with_file_existing"
-    "test_parse_abnormal_specifiers"
-    "test_parse_artifact_metadata"
-    "test_parse_metadata_with_extras"
-    "test_parse_remote_link_metadata"
-    "test_pep582_launcher_for_python_interpreter"
-    "test_rollback_after_commit"
-    "test_run_with_another_project_root"
-    "test_search_package"
-    "test_show_update_hint"
-    "test_sync_only_different"
-    "test_sync_with_index_change"
-    "test_uninstall_commit_rollback"
-    "test_uninstall_with_console_scripts"
-    "test_update_with_prerelease_without_package_argument"
-    "test_url_requirement_is_not_cached"
     # sys.executable and expected executable are different
     "test_set_non_exist_python_path"
-    # editable inside nixpkgs doesn't seem to be a thing
-    "test_vcs_candidate_in_subdirectory"
-    # pdm compares with what looks to be an older version of poetry
-    "test_parse_poetry_project_metadata"
     # pythonfinder isn't aware of nix's python infrastructure
     "test_auto_isolate_site_packages"
     "test_use_invalid_wrapper_python"
     "test_use_wrapper_python"
-    # broken but wouldn't succeed anyway because of the network
-    "test_sync_in_sequential_mode"
     # tries to read/write files without proper permissions
     "test_completion_command"
     "test_plugin_add"
     "test_plugin_list"
     "test_plugin_remove"
-    # unknown breakage, but probably hitting the internet
-    "test_show_package_on_pypi"
     # tries to treat a gzip file as a zipfile and fails
     "test_resolve_local_artifacts"
   ];

[cpcloud]

Please reenable the tests you deemed slow - they are not connecting to the network.

Interesting, I was seeing connection errors from trying to download setuptools in the sandbox.

I went through them one by one, looking at the tracebacks to see what the issue was and in all of those I deemed slow was because there were multiple attempts to connect to pypi to download setuptools.

In any event, glad this worked out.

[cpcloud]

@FRidh I think addressed your comments, mind reviewing again?

[mweinelt]

The patch is fine as an intermediary solution and can be dropped with the next release. I think this is good to go.

[cpcloud]

@mweinelt @jonringer @FRidh A bit confused here, it looks like python-unstable was merged into staging, but I don't see pdm in staging. Should I rebase this onto staging and change the base branch?

[mweinelt]

Yes, the new target is staging. Sorry, but I think we are waiting for @FRidh here.

[cpcloud]

Huzzah! Thanks everyone!