--strict-channel-priority should be used with the conda resolver #12790
Labels
Comments
4 tasks
HadleyKing
added a commit
to biocompute-objects/galaxy
that referenced
this issue
Nov 9, 2022
* fix and slight changes Heading component * fix heading hierarchy Toolbox fix heading hierarchy admin Home add Heading icons to admin Home * fix history import heading * toolBox and historyImport heading adjustments * rename sizes to fit bootstrap undo default heading size changes * admin heading adjustments tool recommendation heading adjustments * various heading level adjustments new hide-element utility class dataset details css refactor to avoid top-gap * various heading level changes * refactor heading levels unify heading sizes * remove headings demo * add headings documentation * fix unit tests * adjust separator heading gap * remove tool card section * remove margin for inline headings * remove toolcard heading css override * Add test for ignored comments * Fix skip XML comments when linting * Add type annotations to ``config_manage.py`` And some code cleanups. * using filter is:deleted to display deleted workflows * creating simple Vue test for restoring deleted WF * passing disabled prop for Tags * adding conditional rendering based on deleted attribute, implementing stubbed out onRestore method in both WFList (just refreshes) and WFDropdown (actually makes the call to restore). onRestore in Dropdown needs to use defined undelete endpoint * removing confirmation message for restoration * connecting to undelete endpoint * fixing test after connecting to endpoint * update the search help box with new filtering option * Test upgrade to bootstrap-vue 2.23. This uses vue-compat to (hopefully?) achieve compatibility with vue3. * Add test for ignored comments * Fix skip XML comments when linting * Fix invocation filtering Fixes galaxyproject#14851 * Rewrite `CondaContext._guess_conda_properties()` to use `conda info` * Require also a recent pyOpenSSL package when installing conda Fix galaxyproject#14848 . * Add `--strict-channel-priority` to conda create/install commands Fix galaxyproject#12790 . This is added only if using conda >=4.7.5 , which includes this bug fix: conda/conda#8819 * Drop confirmation on delete since it's easily revertible now. * Remove jest testing of workflow deletion confirmation (since we don't confirm anymore) * Fix html in helpHtml (missing tags mainly) * Update lib/galaxy/webapps/galaxy/buildapp.py * Drop accept alert from workflow deletion selenium test -- there is no confirmation now * wrap FAI icons in spans, so the tooltip triggers properly * Add breaking tests for container_resolvers index and show * lint * update lib/galaxy/webapps/galaxy/api/container_resolution.py and lib/galaxy/tool_util/deps/views.py * format and change name of class and file * add test for /api/container_resolvers/resolve * update lib/galaxy/tool_util/deps/views.py * Drop broken output.parent logic ⚠ I don't know what this was used for, but none of the dataset-like models have a children list (was probably a relationship at one point). We have no coverage at this point: https://app.codecov.io/github/galaxyproject/galaxy/blob/dev/lib/galaxy/tools/actions/__init__.py and if you add type hints to `out_data` this breaks because there's no children attribute on DatasetInstance. * Use dnspython to check specifically for an "MX" DNS record when validating the domain name of the email address for a new user. Fix galaxyproject#14829 . * Try to fall back to the A record if MX fails * add section indicator to conditionals * Release tool-util package 22.1.3 * Start work on galaxy-tool-util 22.1.4 * ToolSearch: add a front-end Whoosh parser, add ontology search * Fix response encoding when getting tool citations via API Fix the first author surname in: https://usegalaxy.eu/api/tools/lotus2/citations without regressing on galaxyproject#9369 . When decoding the response content, it is better to default to utf-8 than trust the encoding guessed from the content itself. * Add missing ``unittest_utils`` package to galaxy-tool-util Fix: ``` ModuleNotFoundError: No module named 'galaxy.tool_util.unittest_utils' ``` in conda-forge/galaxy-tool-util-feedstock#4 . * Release tool-util package 22.1.4 * Start work on galaxy-tool-util 22.1.5 * Enable attempt restricted values for input parameter connected to subworkflow input * Inject modules & connections, then build runtime state for all steps That means steps can reason about connections that otherwise wouldn't have modules attached when building runtime inputs. * Add jest test for tool search utilities.js * ToolSearch: Conform to style changes in headings * ToolSearch: ToolBox header style changes * reintroduce decoding for PUT /dataset/set_edit calls * Move model/mapping utils into packaged code for reuse by the tool shed. * Always inject all modules, then compute runtime state * Explicitly set test data location for converter tests * Simplify head wrapper - same functionality, no pipe * Fix loading subworkflows from editor interface This probably broke in galaxyproject#13984. There were 3 issues here: Subworkflows are referenced by Workflow.id, not StoredWorkflow.id, so we'd build the wrong url. The other issue is that if we push the new url into the router nothing happens as we re-use the component. Instead we need to listen to route changes and reload the editor config. The third problem is that the main editor component loads the workflow data only in the created hook. I think this needs further refactoring to not use the controller endpoint; everything is available through exisiting APIs and can be cached (toolbox, workflows, tags), but this fixes subworkflow editing for now. * Use biocontainers * Simplify lint and test command now that planemo supports multiple test artefact paths * Adds href to collection download link so users have the ability to copy it. * Bump wrapper version and declare updates to it WF-safe * Install latest miniconda and conda * Fix mistakenly overwriting ``self._conda_version`` with conda-build version. Introduced in commit 0ed6664 . * Retry conda `install` and `create` commands without strict channel priority Restore the possibility to solve environments with packages present in multiple channels but whose requested version is not available in the highest priority one. The issue applies only when using conda >=4.7.5. Fix the following error for the `test/integration/test_resolvers.py::TestCondaResolutionIntegration::test_dependency_install` integration test: ``` ____________ TestCondaResolutionIntegration.test_dependency_install ____________ self = <integration.test_resolvers.TestCondaResolutionIntegration object at 0x7f7009682950> def test_dependency_install(self): """ Test installation of GNUPLOT dependency. """ data = GNUPLOT create_response = self._post("dependency_resolvers/dependency", data=data, admin=True) self._assert_status_code_is(create_response, 200) response = create_response.json() > self._assert_dependency_type(response) test/integration/test_resolvers.py:46: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <integration.test_resolvers.TestCondaResolutionIntegration object at 0x7f7009682950> response = {'cacheable': False, 'dependency_resolver': {'auto_init': True, 'auto_install': False, 'can_uninstall_dependencies': T...nda,defaults', ...}, 'dependency_type': 'conda', 'environment_path': '/tmp/tmp5f_37kj2/conda/envs/__gnuplot@_uv_', ...} type = 'conda', exact = True def _assert_dependency_type(self, response, type="conda", exact=True): if "dependency_type" not in response: raise Exception(f"Response [{response}] did not contain key 'dependency_type'") dependency_type = response["dependency_type"] assert dependency_type == type, f"Dependency type [{dependency_type}] not the expected value [{type}]" if "exact" not in response: raise Exception(f"Response [{response}] did not contain key 'exact'") > assert response["exact"] is exact E assert False is True test/integration/test_resolvers.py:150: AssertionError ``` due to the fact that this test tries to install version 4.6 of gnuplot, while in the highest priority conda-forge channel the oldest version available is 5.0.5. When trying to create the environment, conda fails to resolve it, so Galaxy falls back to create an environment with an unversioned gnuplot, which fails the exact version test. Introduced in commit bb000b2 . * Force recomputation of conda properties after conda `install` conda version and conda-build availability may change after an `install` in the root environment. * Add type annotation to ``conda_util.py`` Also: - Small refactorings * Install and use libmamba solver * Drop converter variable Co-authored-by: Nicola Soranzo <[email protected]> * Drop more unused stuff * Don't do biocontainers for now * Fix mistakenly overwriting ``self._conda_version`` with conda-build version. Introduced in commit 0ed6664 . * Retry conda `install` and `create` commands without strict channel priority Restore the possibility to solve environments with packages present in multiple channels but whose requested version is not available in the highest priority one. The issue applies only when using conda >=4.7.5. Fix the following error for the `test/integration/test_resolvers.py::TestCondaResolutionIntegration::test_dependency_install` integration test: ``` ____________ TestCondaResolutionIntegration.test_dependency_install ____________ self = <integration.test_resolvers.TestCondaResolutionIntegration object at 0x7f7009682950> def test_dependency_install(self): """ Test installation of GNUPLOT dependency. """ data = GNUPLOT create_response = self._post("dependency_resolvers/dependency", data=data, admin=True) self._assert_status_code_is(create_response, 200) response = create_response.json() > self._assert_dependency_type(response) test/integration/test_resolvers.py:46: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <integration.test_resolvers.TestCondaResolutionIntegration object at 0x7f7009682950> response = {'cacheable': False, 'dependency_resolver': {'auto_init': True, 'auto_install': False, 'can_uninstall_dependencies': T...nda,defaults', ...}, 'dependency_type': 'conda', 'environment_path': '/tmp/tmp5f_37kj2/conda/envs/__gnuplot@_uv_', ...} type = 'conda', exact = True def _assert_dependency_type(self, response, type="conda", exact=True): if "dependency_type" not in response: raise Exception(f"Response [{response}] did not contain key 'dependency_type'") dependency_type = response["dependency_type"] assert dependency_type == type, f"Dependency type [{dependency_type}] not the expected value [{type}]" if "exact" not in response: raise Exception(f"Response [{response}] did not contain key 'exact'") > assert response["exact"] is exact E assert False is True test/integration/test_resolvers.py:150: AssertionError ``` due to the fact that this test tries to install version 4.6 of gnuplot, while in the highest priority conda-forge channel the oldest version available is 5.0.5. When trying to create the environment, conda fails to resolve it, so Galaxy falls back to create an environment with an unversioned gnuplot, which fails the exact version test. Introduced in commit bb000b2 . * Force recomputation of conda properties after conda `install` conda version and conda-build availability may change after an `install` in the root environment. * catch two possible specific exceptions before throwing 500 * Add ludwig report datatype * Drop nose It's deprecated and unmaintained, and is broken under Python 3.10 . * Use ``galaxy_driver_class`` to create tool shed driver Use ``autouse`` fixture to configure test driver * Use local import of sqlalchemy sessions If we import too early we just get the inital None. Slightly more robust but still bad. * Drop scripts/function_tests.py * Update cryptography to 38.0.3 * remove the whole trycatch block exceptions should be handled by fastapi * Progress toward fixing extended metadata with Pulsar * refactor FormElement * move style to component * fix tests * deprecate attribute props * fix showField * fix test * add required and optional hints * add test cases for required and optional form elements * make required mark inline in workflow tool forms * remove unused styles * remove unused css class .icon * fix isEmpty for edge cases * add password type to isEmpty string check * fix title display * Consistently serialize tags in Visualization * Restore rating stars and display for published items * Add email to sharable serializer * Add hashed email for gravatar to serializer * Restore gravater in vueified published items * Fix header class * Always call strip() on data_column column values Not just when they start with `c`. Should fix running workflows that were created by manually writing columns in the text area field and then hitting enter. * Drop explicit test-data argument Co-authored-by: Nicola Soranzo <[email protected]> * Remove ability to delete individual files from tool shed. Working toward a simplified, Planemo-used API that is just uploading archives. * Update interactivetool_ml_jupyter_notebook.xml * Fix server errors when querying for invalid repos or repos without valid tools * Use _run_jobs instead of _run_workflows * Use dnspython to check specifically for an "MX" DNS record when validating the domain name of the email address for a new user. Fix galaxyproject#14829 . * Try to fall back to the A record if MX fails * Fix copy_sample_file usage * Fix displaying named tags * Upgrade vue-router to latest 3x series for composable * Add missing `dnspython` dependency in 22.05 * Mark non-slow unit tests with more relevant ``@skip_if_github_down`` decorator * Skip S3 unit tests conditionally on s3fs module presence * Replace ``GALAXY_TEST_INCLUDE_SLOW`` env variable with ``external_dependency_management`` pytest marker Fix mulled tests missing tests in `test/unit/tool_util/test_conda_resolution.py` . Also: - Add `coverage` tox factor to generate coverage report. * Move md5 utility to util * Fix the ``singularity_container_test()`` function Fix ``test_singularity_container_test`` unit test, which failed with: ``` test/unit/tool_util/mulled/test_mulled_update_singularity_containers.py:48: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ lib/galaxy/tool_util/deps/mulled/mulled_update_singularity_containers.py:62: in singularity_container_test check_output(exec_command.extend(["bash", "-c", test_command]), stderr=subprocess.STDOUT) /usr/lib/python3.8/subprocess.py:415: in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, /usr/lib/python3.8/subprocess.py:493: in run with Popen(*popenargs, **kwargs) as process: /usr/lib/python3.8/subprocess.py:858: in __init__ self._execute_child(args, executable, preexec_fn, close_fds, > args = list(args) E TypeError: 'NoneType' object is not iterable /usr/lib/python3.8/subprocess.py:1569: TypeError ``` Broken in commit 1dc1e5f . Also: - Use `tmp_path` pytest fixture. - Add type annotations. - Code refactorings. - Install singularity in the mulled GitHub workflow so that the 2 singularity unit tests are not skipped any more. * Fix conda-build version not compatible with the new miniconda Python version introduced in commit 06832df . Also: - Add unit test. - Use `tmp_path` pytest fixture. - Test code refactoring. * fix button hidden from anonymous users * Update migration instructions * Remove rating from published items * Strip item rating helper from controller endpoints * Remove item rating from legacy mako * Change custos <url> in oidc_backends_config * Fix download urls in embedded workflow component * Add workflow display component test * Fix broken subworkflow reodering on workflow import We started reordering workflow steps on import in galaxyproject#13641. If subworkflow steps are reordered before the input connections are set the wrong subworkflow steps are referenced when we build the workflow connection. This broke the IWC tests for gromacs. We now do not reorder subworkflow steps until the parent workflow is being reordered, which is after all connections are set up. * Revert "Use _run_jobs instead of _run_workflows" This reverts commit a01c8c5. _run_workflow has the right type annotations, but 21.09 didn't have this. * Adjust for disabled history edit toggle * Replace ``unittest.TestCase`` in API, framework, performance, Selenium and ToolShed tests In API tests, replace no more working `uses_test_history` decorator (because the `history_id` method parameter is interpreted as a fixture) with a real class-level pytest fixture. As a consequence, replace use of `self.history_id` in API tests because it interferes with the class-level `history_id` fixture. Also: - Some type annotations * Remove unused ``require_new`` and ``GALAXY_TEST_HISTORY_ID`` * Remove outdated check scripts * Add GitHub workflow to check for regression in test class names * Rename class which is not derived from ``TestCase`` to make `.ci/check_test_class_names.sh` happy. * Fix select2 value selection in selenium * Fix minor typo in text case * Fix merge forward of galaxyproject#14918 The new test needs to use the `history_id` fixture and pass it to `self._run_workflow()` . * Add ``require_new_history`` marker to ``pytest.ini`` * Stabilize test_edit_subworkflow selenium test The click positions don't seem to be lined up. Will probably go away with the workflow editor update I'm working on, in the meantime xfail (which turns into xpass when passing https://docs.pytest.org/en/7.1.x/how-to/skipping.html) will do the trick. * Remove incorrect references to datasets instead of dataset instances in run_request.py. See Marius' notes on https://github.com/galaxyproject/galaxy/pull/14938/files. * Fix deprecation of ``ProcessPool.schedule()`` since pebble 5.0 Fix: ``` DeprecationWarning: schedule is deprecated; use submit instead ``` Also: - Add some type annotations * Add types to galaxy/util/zipstream.py. Brought in from Marius' default file work. * More type annotations * Order steps relation of workflow_invocation by order_index * small change to job runner docs to state more explicitly what `check_watched_item()` should return * Fix indentation * Fix all B027 errors reported by new flake8-bugbear * Small refactoring * Fix pytest warning Fix: ``` PytestReturnNotNoneWarning: Expected None, but packages/tool_util/tests/tool_util/test_verify.py::test_file_list returned [TestFile(value=b'A\nB\nC', path='/tmp/tmpi72pz4ur.txt'), TestFile(value=...], which will be an error in a future version of pytest. Did you mean to use `assert` instead of `return`? ``` * Cleanup test case a bit. Co-authored-by: Laila Los <[email protected]> Co-authored-by: davelopez <[email protected]> Co-authored-by: Marius van den Beek <[email protected]> Co-authored-by: Nicola Soranzo <[email protected]> Co-authored-by: Assunta DeSanto <[email protected]> Co-authored-by: Martin Cech <[email protected]> Co-authored-by: Dannon Baker <[email protected]> Co-authored-by: Aysam Guerler <[email protected]> Co-authored-by: cat-bro <[email protected]> Co-authored-by: John Chilton <[email protected]> Co-authored-by: Ahmed Awan <[email protected]> Co-authored-by: Wolfgang Maier <[email protected]> Co-authored-by: Nicola Soranzo <[email protected]> Co-authored-by: John Davis <[email protected]> Co-authored-by: qiagu <[email protected]> Co-authored-by: Nate Coraor <[email protected]> Co-authored-by: Anup Kumar <[email protected]> Co-authored-by: Alireza Heidari <[email protected]> Co-authored-by: Simon Bray <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Related to #12788
Bioconda best practices indicate that strict channel priority should be used: bioconda/bioconda-recipes#13774
However this is not implemented in the conda resolver for galaxy yet. The difference between having this setting and not can be 15 seconds vs 3hours+ (10000+ seconds). Multiple orders of magnitude slower solving.
Preferably strict channel priority should always be set.
The text was updated successfully, but these errors were encountered: