Conda core requirements install should use channels from env

[vidartf]

While troubleshooting / looking for a workaround for the issue in #5545, it came to my attention that several of the conda packages that are missing the setuptools metadata come from the default anaconda channel. The same packages on the conda-forge channel will normally include this metadata. This caused me some confusion, as I had specified the channels in my environment.yml to only use conda-forge. Looking at the build log, I see that the install_core_requirements step always use the default channel. This will cause some packages to be installed/overwritten from the default channel.

Is it possible to have this command use the same channels as those in environment.yml? Possibly with default as the lowest priority channel?

I would be willing to write such a PR if the consensus is that this is OK :)

[stsewd]

Not sure about this, this will be tricky to know which channel to use, some channels probably don't include some core requirements and builds will fail.

[vidartf]

How about something like this:

env = read_yaml(self.config.conda.environment)
channels = env.get('channels', [])
if 'default' not in channels:
    channels.append('default')

cmd = [
    'conda',
    'install',
    '--yes',
    '--quiet',
    '--name',
]
# TODO: Wrap iter logic in util function:
cmd.extend(
    itertools.chain.from_iterable(zip(
        ['-c'] * len(channels), 
        channels
)))

cmd.append(self.version.slug)

...

[chrisjsewell]

+1 this issue.

At a bare minimum, I think the install_core_requirements step should include -c conda-forge in cmd.

Ideally though, I would like the option to completely skip this step, and handle the install of the core requirements (mock, pillow, sphinx, sphinx_rtd_theme) in my environment.yaml. This would be particularly helpful if, for example, one requires a particular version of sphinx.

Something like:

.readthedocs.yml

version: 2

conda:
  environment: docs/environment.yml
  install_core: true

docs/environment.yml

channels:
  - conda-forge
dependencies:
  - python =3.6
  - mock
  - pillow
  - sphinx =1.8
  - sphinx_rtd_theme

[humitos]

This issue is related to #3829 and not being able to pin packages at env creation because they will be upgraded in the second step when RTD installs its needed dependencies.

[humitos]

Hi @vidartf cc @chrisjsewell

Is it possible to have this command use the same channels as those in environment.yml?

Yes. We do have a Feature Flag now that I can enable in your project if you want. This flag makes RTD to install all the dependencies in just one step using your own environment.yml file only (RTD dependencies are appended to the YAML file).

The flag is called: CONDA_APPEND_CORE_REQUIREMENTS

See https://docs.readthedocs.io/en/latest/guides/feature-flags.html

Let me know if you want me to add this flag to your project.

[vidartf]

@humitos This sounds promising! Would you mind enabling the flag on https://readthedocs.org/projects/ipyscales/ for now? Then if/when everything works smoothly I probably have a few more (~5 projects I think) if possible.

[humitos]

@vidartf done! I triggered a new build at https://readthedocs.org/projects/ipyscales/builds/9574163/

Note that there is only one conda create command and before that one, you can see the cat command that shows the content of the environment after appending the packages.

Feel free to close this issue if this is working as you are expecting.

[no-response]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further. Thanks!

[vidartf]

@humitos I was finally able to try this out (fixing some other issues), but immediately ran into an issue:

> conda env create --quiet --name update-support --file docs/environment.yml

Collecting package metadata (repodata.json): ...working... Killed


Command killed due to excessive memory consumption

[stsewd]

@vidartf please check https://docs.readthedocs.io/en/stable/guides/build-using-too-many-resources.html

[vidartf]

@stsewd This is conda install that is triggering the "out of resources", not my build. Would it be possible to add a flag for using mamba instead of conda for the install step?

[vidartf]

Note, the dependencies listed in my conda env are also quite limited:

• python=3.5
• nodejs
• numpy
• sphinx
• nbsphinx
• pip

If these are enough to trip the conda install, this flag (CONDA_APPEND_CORE_REQUIREMENTS) will probably never work?

[vidartf]

Seem like adding - nodefaults to the channels list in the environment file helps conda come within limits again. 👍

[stsewd]

@vidartf great, let us know if you still run out of resources.

[vidartf]

With builds now passing, it would be wonderful if the following projects could have the CONDA_APPEND_CORE_REQUIREMENTS option enabled as well:

• ipycombobox
• ipydatetime

[stsewd]

@vidartf done, let me know if you have any problem. Probably you should wipe the environment before triggering a new build on those projects https://docs.readthedocs.io/en/stable/guides/wipe-environment.html